Bagging and boosting are two powerful ensemble techniques in machine learning – they are must-knows for data scientists! After reading this article, you are going to have a solid understanding of how bagging and boosting work and when to use them. We’ll cover the following topics, relying heavily on examples to give hands-on illustration of the key concepts: How Ensembling helps create powerful models Bagging: Adding stability to ML models Boosting: Reducing bias in weak learners Bagging vs. Boosting – when to use each and why Creating powerful models with ensembling In Machine Learning, ensembling is a broad term that refers to any technique that creates predictions by combining the predictions from multiple models. If there is more than one model involved in making a prediction, the technique is using ensembling! Ensembling approaches can often improve the performance of a single model. Ensembling can help reduce: Variance by averaging multiple models Bias by iteratively improving on errors Overfitting because using multiple models can increase robustness to spurious relationships Bagging and boosting are both ensemble methods that can perform much better than their single-model counterparts. Let’s get into the details of these now! Bagging: Adding stability to ML models Bagging is a specific ensembling technique that is used to reduce the variance of a predictive model. Here, I’m talking about variance in the machine learning sense – i.e., how much a model varies with changes to the training dataset – not variance in the statistical sense which measures the spread of a distribution. Because bagging helps reduce an ML model’s variance, it will often improve models that are high variance (e.g., decision trees and KNN) but won’t do much good for models that are low variance (e.g., linear regression). Now that we understand when bagging helps (high variance models), let’s get into the details of the inner workings to understand how it helps! The bagging algorithm is iterative in nature – it builds multiple models by repeating the following three steps: Bootstrap a dataset from the original training data Train a model on the bootstrapped dataset Save the trained model The collection of models created in this process is called an ensemble. When it is time to make a prediction, each model in the ensemble makes its own prediction – the final bagged prediction is the average (for regression) or majority vote (for classification) of all of the ensemble’s predictions. Now that we understand how bagging works, let’s take a few minutes to build an intuition for why it works. We’ll borrow a familiar idea from traditional statistics: sampling to estimate a population mean. In statistics, each sample drawn from a distribution is a random variable. Small sample sizes tend to have high variance and may provide poor estimates of the true mean. But as we collect more samples, the average of those samples becomes a much better approximation of the population mean. Similarly, we can think of each of our individual decision trees as a random variable — after all, each tree is trained on a different random sample of the data! By averaging predictions from many trees, bagging reduces variance and produces an ensemble model that better captures the true relationships in the data. Bagging Example We will be using the load_diabetes1 dataset from the scikit-learn Python package to illustrate a simple bagging example. The dataset has 10 input variables – Age, Sex, BMI, Blood Pressure and 6 blood serum levels (S1-S6). And a single output variable that is a measurement of disease progression. The code below pulls in our data and does some very simple cleaning. With our dataset established, let’s start modeling! # pull in and format data from sklearn.datasets import load_diabetes diabetes = load_diabetes(as_frame=True) df = pd.DataFrame(diabetes.data, columns=diabetes.feature_names) df.loc[:, 'target'] = diabetes.target df = df.dropna() For our example, we will use basic decision trees as our base models for bagging. Let’s first verify that our decision trees are indeed high variance. We will do this by training three decision trees on different bootstrapped datasets and observing the variance of the predictions for a test dataset. The graph below shows the predictions of three different decision trees on the same test dataset. Each dotted vertical line is an individual observation from the test dataset. The three dots on each line are the predictions from the three different decision trees. Variance of decision trees on test data points – image by author In the chart above, we see that individual trees can give very different predictions (spread of the three dots on each vertical line) when trained on bootstrapped datasets. This is the variance we have been talking about! Now that we see that our trees aren’t very robust to training samples – let’s average the predictions to see how bagging can help! The chart below shows the average of the three trees. The diagonal line represents perfect predictions. As you can see, with bagging, our points are tighter and more centered around the diagonal. image by author We’ve already seen significant improvement in our model with the average of just three trees. Let’s beef up our bagging algorithm with more trees! Here is the code to bag as many trees as we want: def train_bagging_trees(df, target_col, pred_cols, n_trees): ''' Creates a decision tree bagging model by training multiple decision trees on bootstrapped data. inputs df (pandas DataFrame) : training data with both target and input columns target_col (str) : name of target column pred_cols (list) : list of predictor column names n_trees (int) : number of trees to be trained in the ensemble output: train_trees (list) : list of trained trees ''' train_trees = [] for i in range(n_trees): # bootstrap training data temp_boot = bootstrap(train_df) #train tree temp_tree = plain_vanilla_tree(temp_boot, target_col, pred_cols) # save trained tree in list train_trees.append(temp_tree) return train_trees def bagging_trees_pred(df, train_trees, target_col, pred_cols): ''' Takes a list of bagged trees and creates predictions by averaging the predictions of each individual tree. inputs df (pandas DataFrame) : training data with both target and input columns train_trees (list) : ensemble model - which is a list of trained decision trees target_col (str) : name of target column pred_cols (list) : list of predictor column names output: avg_preds (list) : list of predictions from the ensembled trees ''' x = df[pred_cols] y = df[target_col] preds = [] # make predictions on data with each decision tree for tree in train_trees: temp_pred = tree.predict(x) preds.append(temp_pred) # get average of the trees' predictions sum_preds = [sum(x) for x in zip(*preds)] avg_preds = [x / len(train_trees) for x in sum_preds] return avg_preds The functions above are very simple, the first trains the bagging ensemble model, the second takes the ensemble (simply a list of trained trees) and makes predictions given a dataset. With our code established, let’s run multiple ensemble models and see how our out-of-bag predictions change as we increase the number of trees. Out-of-bag predictions vs. actuals colored by number of bagged trees – image by author Admittedly, this chart looks a little crazy. Don’t get too bogged down with all of the individual data points, the lines dashed tell the main story! Here we have 1 basic decision tree model and 3 bagged decision tree models – with 3, 50 and 150 trees. The color-coded dotted lines mark the upper and lower ranges for each model’s residuals. There are two main takeaways here: (1) as we add more trees, the range of the residuals shrinks and (2) there is diminishing returns to adding more trees – when we go from 1 to 3 trees, we see the range shrink a lot, when we go from 50 to 150 trees, the range tightens just a little. Now that we’ve successfully gone through a full bagging example, we are about ready to move onto boosting! Let’s do a quick overview of what we covered in this section: Bagging reduces variance of ML models by averaging the predictions of multiple individual models Bagging is most helpful with high-variance models The more models we bag, the lower the variance of the ensemble – but there are diminishing returns to the variance reduction benefit Okay, let’s move on to boosting! Boosting: Reducing bias in weak learners With bagging, we create multiple independent models – the independence of the models helps average out the noise of individual models. Boosting is also an ensembling technique; similar to bagging, we will be training multiple models…. But very different from bagging, the models we train will be dependent. Boosting is a modeling technique that trains an initial model and then sequentially trains additional models to improve the predictions of prior models. The primary target of boosting is to reduce bias – though it can also help reduce variance. We’ve established that boosting iteratively improves predictions – let’s go deeper into how. Boosting algorithms can iteratively improve model predictions in two ways: Directly predicting the residuals of the last model and adding them to the prior predictions – think of it as residual corrections Adding more weight to the observations that the prior model predicted poorly Because boosting’s main goal is to reduce bias, it works well with base models that typically have more bias (e.g., shallow decision trees). For our examples, we are going to use shallow decision trees as our base model – we will only cover the residual prediction approach in this article for brevity. Let’s jump into the boosting example! Predicting prior residuals The residuals prediction approach starts off with an initial model (some algorithms provide a constant, others use one iteration of the base model) and we calculate the residuals of that initial prediction. The second model in the ensemble predicts the residuals of the first model. With our residual predictions in-hand, we add the residual predictions to our initial prediction (this gives us residual corrected predictions) and recalculate the updated residuals…. we continue this process until we have created the number of base models we specified. This process is pretty simple, but is a little hard to explain with just words – the flowchart below shows a simple, 4-model boosting algorithm. Flowchart of simple, 4 model boosting algorithm – image by author When boosting, we need to set three main parameters: (1) the number of trees, (2) the tree depth and (3) the learning rate. I’ll spend a little time discussing these inputs now. Number of Trees For boosting, the number of trees means the same thing as in bagging – i.e., the total number of trees that will be trained for the ensemble. But, unlike boosting, we should not err on the side of more trees! The chart below shows the test RMSE against the number of trees for the diabetes dataset. Unlike with bagging, too many trees in boosting leads to overfitting! – image by author This shows that the test RMSE drops quickly with the number of trees up until about 200 trees, then it starts to creep back up. It looks like a classic ‘overfitting’ chart – we reach a point where more trees becomes worse for the model. This is a key difference between bagging and boosting – with bagging, more trees eventually stop helping, with boosting more trees eventually start hurting! With bagging, more trees eventually stops helping, with boosting more trees eventually starts hurting! We now know that too many trees are bad, and too few trees are bad as well. We will use hyperparameter tuning to select the number of trees. Note – hyperparameter tuning is a huge subject and way outside of the scope of this article. I’ll demonstrate a simple grid search with a train and test dataset for our example a little later. Tree Depth This is the maximum depth for each tree in the ensemble. With bagging, trees are often allowed to go as deep they want because we are looking for low bias, high variance models. With boosting however, we use sequential models to address the bias in the base learners – so we aren’t as concerned about generating low-bias trees. How do we decide how the maximum depth? The same technique that we’ll use with the number of trees, hyperparameter tuning. Learning Rate The number of trees and the tree depth are familiar parameters from bagging (although in bagging we often didn’t put a limit on the tree depth) – but this ‘learning rate’ character is a new face! Let’s take a moment to get familiar. The learning rate is a number between 0 and 1 that is multiplied by the current model’s residual predictions before it is added to the overall predictions. Here’s a simple example of the prediction calculations with a learning rate of 0.5. Once we understand the mechanics of how the learning rate works, we will discuss the why the learning rate is important. The learning rate discounts the residual prediction before updating the actual target prediction – image by author So, why would we want to ‘discount’ our residual predictions, wouldn’t that make our predictions worse? Well, yes and no. For a single iteration, it will likely make our predictions worse – but, we are doing multiple iterations. For multiple iterations, the learning rate keeps the model from overreacting to a single tree’s predictions. It will probably make our current predictions worse, but don’t worry, we will go through this process multiple times! Ultimately, the learning rate helps mitigate overfitting in our boosting model by lowering the influence of any single tree in the ensemble. You can think of it as slowly turning the steering wheel to correct your driving rather than jerking it. In practice, the number of trees and the learning rate have an opposite relationship, i.e., as the learning rate goes down, the number of trees goes up. This is intuitive, because if we only allow a small amount of each tree’s residual prediction to be added to the overall prediction, we are going to need a lot more trees before our overall prediction will start looking good. Ultimately, the learning rate helps mitigate overfitting in our boosting model by lowering the influence of any single tree in the ensemble. Alright, now that we’ve covered the main inputs in boosting, let’s get into the Python coding! We need a couple of functions to create our boosting algorithm: Base decision tree function – a simple function to create and train a single decision tree. We will use the same function from the last section called ‘plain_vanilla_tree.’ Boosting training function – this function sequentially trains and updates residuals for as many decision trees as the user specifies. In our code, this function is called ‘boost_resid_correction.’ Boosting prediction function – this function takes a series of boosted models and makes final ensemble predictions. We call this function ‘boost_resid_correction_pred.’ Here are the functions written in Python: # same base tree function as in prior section def plain_vanilla_tree(df_train, target_col, pred_cols, max_depth = 3, weights=[]): X_train = df_train[pred_cols] y_train = df_train[target_col] tree = DecisionTreeRegressor(max_depth = max_depth, random_state=42) if weights: tree.fit(X_train, y_train, sample_weights=weights) else: tree.fit(X_train, y_train) return tree # residual predictions def boost_resid_correction(df_train, target_col, pred_cols, num_models, learning_rate=1, max_depth=3): ''' Creates boosted decision tree ensemble model. Inputs: df_train (pd.DataFrame) : contains training data target_col (str) : name of target column pred_col (list) : target column names num_models (int) : number of models to use in boosting learning_rate (float, def = 1) : discount given to residual predictions takes values between (0, 1] max_depth (int, def = 3) : max depth of each tree model Outputs: boosting_model (dict) : contains everything needed to use model to make predictions - includes list of all trees in the ensemble ''' # create initial predictions model1 = plain_vanilla_tree(df_train, target_col, pred_cols, max_depth = max_depth) initial_preds = model1.predict(df_train[pred_cols]) df_train['resids'] = df_train[target_col] - initial_preds # create multiple models, each predicting the updated residuals models = [] for i in range(num_models): temp_model = plain_vanilla_tree(df_train, 'resids', pred_cols) models.append(temp_model) temp_pred_resids = temp_model.predict(df_train[pred_cols]) df_train['resids'] = df_train['resids'] - (learning_rate*temp_pred_resids) boosting_model = {'initial_model' : model1, 'models' : models, 'learning_rate' : learning_rate, 'pred_cols' : pred_cols} return boosting_model # This function takes the residual boosted model and scores data def boost_resid_correction_predict(df, boosting_models, chart = False): ''' Creates predictions on a dataset given a boosted model. Inputs: df (pd.DataFrame) : data to make predictions boosting_models (dict) : dictionary containing all pertinent boosted model data chart (bool, def = False) : indicates if performance chart should be created Outputs: pred (np.array) : predictions from boosted model rmse (float) : RMSE of predictions ''' # get initial predictions initial_model = boosting_models['initial_model'] pred_cols = boosting_models['pred_cols'] pred = initial_model.predict(df[pred_cols]) # calculate residual predictions from each model and add models = boosting_models['models'] learning_rate = boosting_models['learning_rate'] for model in models: temp_resid_preds = model.predict(df[pred_cols]) pred += learning_rate*temp_resid_preds if chart: plt.scatter(df['target'], pred) plt.show() rmse = np.sqrt(mean_squared_error(df['target'], pred)) return pred, rmse Sweet, let’s make a model on the same diabetes dataset that we used in the bagging section. We’ll do a quick grid search (again, not doing anything fancy with the tuning here) to tune our three parameters and then we’ll train the final model using the boost_resid_correction function. # tune parameters with grid search n_trees = [5,10,30,50,100,125,150,200,250,300] learning_rates = [0.001, 0.01, 0.1, 0.25, 0.50, 0.75, 0.95, 1] max_depths = my_list = list(range(1, 16)) # Create a dictionary to hold test RMSE for each 'square' in grid perf_dict = {} for tree in n_trees: for learning_rate in learning_rates: for max_depth in max_depths: temp_boosted_model = boost_resid_correction(train_df, 'target', pred_cols, tree, learning_rate=learning_rate, max_depth=max_depth) temp_boosted_model['target_col'] = 'target' preds, rmse = boost_resid_correction_predict(test_df, temp_boosted_model) dict_key = '_'.join(str(x) for x in [tree, learning_rate, max_depth]) perf_dict[dict_key] = rmse min_key = min(perf_dict, key=perf_dict.get) print(perf_dict[min_key]) And our winner is — 50 trees, a learning rate of 0.1 and a max depth of 1! Let’s take a look and see how our predictions did. Tuned boosting actuals vs. residuals – image by author While our boosting ensemble model seems to capture the trend reasonably well, we can see off the bat that it isn’t predicting as well as the bagging model. We could probably spend more time tuning – but it could also be the case that the bagging approach fits this specific data better. With that said, we’ve now earned an understanding of bagging and boosting – let’s compare them in the next section! Bagging vs. Boosting – understanding the differences We’ve covered bagging and boosting separately, the table below brings all the information we’ve covered to concisely compare the approaches: image by author Note: In this article, we wrote our own bagging and boosting code for educational purposes. In practice you will just use the excellent code that is available in Python packages or other software. Also, people rarely use ‘pure’ bagging or boosting – it is much more common to use more advanced algorithms that modify the plain vanilla bagging and boosting to improve performance. Wrapping it up Bagging and boosting are powerful and practical ways to improve weak learners like the humble but flexible decision tree. Both approaches use the power of ensembling to address different problems – bagging for variance, boosting for bias. In practice, pre-packaged code is almost always used to train more advanced machine learning models that use the main ideas of bagging and boosting but, expand on them with multiple improvements. I hope that this has been helpful and interesting – happy modeling! Dataset is originally from the National Institute of Diabetes and Digestive and Kidney Diseases and is distributed under the public domain license for use without restriction. The post Strength in Numbers: Ensembling Models with Bagging and Boosting appeared first on Towards Data Science.

In this tutorial, we demonstrate how to construct an automated Knowledge Graph (KG) pipeline using LangGraph and NetworkX. The pipeline simulates a sequence of intelligent agents that collaboratively perform tasks such as data gathering, entity extraction, relation identification, entity resolution, and graph validation. Starting from a user-provided topic, such as “Artificial Intelligence,” the system methodically extracts relevant entities and relationships, resolves duplicates, and integrates the information into a cohesive graphical structure. By visualizing the final knowledge graph, developers and data scientists gain clear insights into complex interrelations among concepts, making this approach highly beneficial for applications in semantic analysis, natural language processing, and knowledge management. !pip install langgraph langchain_core We install two essential Python libraries: LangGraph, which is used for creating and orchestrating agent-based computational workflows, and LangChain Core, which provides foundational classes and utilities for building language model-powered applications. These libraries enable seamless integration of agents into intelligent data pipelines. import re import networkx as nx import matplotlib.pyplot as plt from typing import TypedDict, List, Tuple, Dict, Any from langchain_core.messages import HumanMessage, AIMessage from langgraph.graph import StateGraph, END We import essential libraries to build an automated knowledge graph pipeline. It includes re for regular expression-based text processing, NetworkX and matplotlib for creating and visualizing graphs, TypedDict and typing annotations for structured data handling, and LangGraph along with langchain_core for orchestrating the interaction between AI agents within the workflow. class KGState(TypedDict): topic: str raw_text: str entities: List[str] relations: List[Tuple[str, str, str]] resolved_relations: List[Tuple[str, str, str]] graph: Any validation: Dict[str, Any] messages: List[Any] current_agent: str We define a structured data type, KGState, using Python’s TypedDict. It outlines the schema for managing state across different steps of the knowledge graph pipeline. It includes details like the chosen topic, gathered text, identified entities and relationships, resolved duplicates, the constructed graph object, validation results, interaction messages, and tracking the currently active agent. def data_gatherer(state: KGState) -> KGState: topic = state["topic"] print(f"📚 Data Gatherer: Searching for information about '{topic}'") collected_text = f"{topic} is an important concept. It relates to various entities like EntityA, EntityB, and EntityC. EntityA influences EntityB. EntityC is a type of EntityB." state["messages"].append(AIMessage(content=f"Collected raw text about {topic}")) state["raw_text"] = collected_text state["current_agent"] = "entity_extractor" return state This function, data_gatherer, acts as the first step in the pipeline. It simulates gathering raw text data about a provided topic (stored in state[“topic”]). It then stores this simulated data into state[“raw_text”], adds a message indicating the data collection completion, and updates the pipeline’s state by setting the next agent (entity_extractor) as active. def entity_extractor(state: KGState) -> KGState: print("🔍 Entity Extractor: Identifying entities in the text") text = state["raw_text"] entities = re.findall(r"Entity[A-Z]", text) entities = [state["topic"]] + entities state["entities"] = list(set(entities)) state["messages"].append(AIMessage(content=f"Extracted entities: {state['entities']}")) print(f" Found entities: {state['entities']}") state["current_agent"] = "relation_extractor" return state The entity_extractor function identifies entities from the collected raw text using a simple regular expression pattern that matches terms like “EntityA”, “EntityB”, etc. It also includes the main topic as an entity and ensures uniqueness by converting the list to a set. The extracted entities are stored in the state, an AI message logs the result, and the pipeline advances to the relation_extractor agent. def relation_extractor(state: KGState) -> KGState: print("🔗 Relation Extractor: Identifying relationships between entities") text = state["raw_text"] entities = state["entities"] relations = [] relation_patterns = [ (r"([A-Za-z]+) relates to ([A-Za-z]+)", "relates_to"), (r"([A-Za-z]+) influences ([A-Za-z]+)", "influences"), (r"([A-Za-z]+) is a type of ([A-Za-z]+)", "is_type_of") ] for e1 in entities: for e2 in entities: if e1 != e2: for pattern, rel_type in relation_patterns: if re.search(f"{e1}.*{rel_type}.*{e2}", text.replace("_", " "), re.IGNORECASE) or \ re.search(f"{e1}.*{e2}", text, re.IGNORECASE): relations.append((e1, rel_type, e2)) state["relations"] = relations state["messages"].append(AIMessage(content=f"Extracted relations: {relations}")) print(f" Found relations: {relations}") state["current_agent"] = "entity_resolver" return state The relation_extractor function detects semantic relationships between entities within the raw text. It uses predefined regex patterns to identify phrases like “influences” or “is a type of” between entity pairs. When a match is found, it adds the corresponding relation as a triple (subject, predicate, object) to the relations list. These extracted relations are stored in the state, a message is logged for agent communication, and control moves to the next agent: entity_resolver. def entity_resolver(state: KGState) -> KGState: print("🔄 Entity Resolver: Resolving duplicate entities") entity_map = {} for entity in state["entities"]: canonical_name = entity.lower().replace(" ", "_") entity_map[entity] = canonical_name resolved_relations = [] for s, p, o in state["relations"]: s_resolved = entity_map.get(s, s) o_resolved = entity_map.get(o, o) resolved_relations.append((s_resolved, p, o_resolved)) state["resolved_relations"] = resolved_relations state["messages"].append(AIMessage(content=f"Resolved relations: {resolved_relations}")) state["current_agent"] = "graph_integrator" return state The entity_resolver function standardizes entity names to avoid duplication and inconsistencies. It creates a mapping (entity_map) by converting each entity to lowercase and replacing spaces with underscores. Then, this mapping is applied to all subjects and objects in the extracted relations to produce resolved relations. These normalized triples are added to the state, a confirmation message is logged, and control is passed to the graph_integrator agent. def graph_integrator(state: KGState) -> KGState: print("📊 Graph Integrator: Building the knowledge graph") G = nx.DiGraph() for s, p, o in state["resolved_relations"]: if not G.has_node(s): G.add_node(s) if not G.has_node(o): G.add_node(o) G.add_edge(s, o, relation=p) state["graph"] = G state["messages"].append(AIMessage(content=f"Built graph with {len(G.nodes)} nodes and {len(G.edges)} edges")) state["current_agent"] = "graph_validator" return state The graph_integrator function constructs the actual knowledge graph using networkx.DiGraph() supports directed relationships. It iterates over the resolved triples (subject, predicate, object), ensures both nodes exist, and then adds a directed edge with the relation as metadata. The resulting graph is saved in the state, a summary message is appended, and the pipeline transitions to the graph_validator agent for final validation. def graph_validator(state: KGState) -> KGState: print("✅ Graph Validator: Validating knowledge graph") G = state["graph"] validation_report = { "num_nodes": len(G.nodes), "num_edges": len(G.edges), "is_connected": nx.is_weakly_connected(G) if G.nodes else False, "has_cycles": not nx.is_directed_acyclic_graph(G) if G.nodes else False } state["validation"] = validation_report state["messages"].append(AIMessage(content=f"Validation report: {validation_report}")) print(f" Validation report: {validation_report}") state["current_agent"] = END return state The graph_validator function performs a basic health check on the constructed knowledge graph. It compiles a validation report containing the number of nodes and edges, whether the graph is weakly connected (i.e., every node is reachable if direction is ignored), and whether the graph contains cycles. This report is added to the state and logged as an AI message. Once validation is complete, the pipeline is marked as finished by setting the current_agent to END. def router(state: KGState) -> str: return state["current_agent"] def visualize_graph(graph): plt.figure(figsize=(10, 6)) pos = nx.spring_layout(graph) nx.draw(graph, pos, with_labels=True, node_color='skyblue', node_size=1500, font_size=10) edge_labels = nx.get_edge_attributes(graph, 'relation') nx.draw_networkx_edge_labels(graph, pos, edge_labels=edge_labels) plt.title("Knowledge Graph") plt.tight_layout() plt.show() The router function directs the pipeline to the next agent based on the current_agent field in the state. Meanwhile, the visualize_graph function uses matplotlib and networkx to display the final knowledge graph, showing nodes, edges, and labeled relationships for intuitive visual understanding. def build_kg_graph(): workflow = StateGraph(KGState) workflow.add_node("data_gatherer", data_gatherer) workflow.add_node("entity_extractor", entity_extractor) workflow.add_node("relation_extractor", relation_extractor) workflow.add_node("entity_resolver", entity_resolver) workflow.add_node("graph_integrator", graph_integrator) workflow.add_node("graph_validator", graph_validator) workflow.add_conditional_edges("data_gatherer", router, {"entity_extractor": "entity_extractor"}) workflow.add_conditional_edges("entity_extractor", router, {"relation_extractor": "relation_extractor"}) workflow.add_conditional_edges("relation_extractor", router, {"entity_resolver": "entity_resolver"}) workflow.add_conditional_edges("entity_resolver", router, {"graph_integrator": "graph_integrator"}) workflow.add_conditional_edges("graph_integrator", router, {"graph_validator": "graph_validator"}) workflow.add_conditional_edges("graph_validator", router, {END: END}) workflow.set_entry_point("data_gatherer") return workflow.compile() The build_kg_graph function defines the complete knowledge graph workflow using LangGraph. It sequentially adds each agent as a node, from data collection to graph validation, and connects them through conditional transitions based on the current agent. The entry point is set to data_gatherer, and the graph is compiled into an executable workflow that guides the automated pipeline from start to finish. def run_knowledge_graph_pipeline(topic): print(f"🚀 Starting knowledge graph pipeline for: {topic}") initial_state = { "topic": topic, "raw_text": "", "entities": [], "relations": [], "resolved_relations": [], "graph": None, "validation": {}, "messages": [HumanMessage(content=f"Build a knowledge graph about {topic}")], "current_agent": "data_gatherer" } kg_app = build_kg_graph() final_state = kg_app.invoke(initial_state) print(f"✨ Knowledge graph construction complete for: {topic}") return final_state The run_knowledge_graph_pipeline function initializes the pipeline by setting up an empty state dictionary with the provided topic. It builds the workflow using build_kg_graph(), then runs it by invoking the compiled graph with the initial state. As each agent processes the data, the state evolves, and the final result contains the complete knowledge graph, validated and ready for use. if __name__ == "__main__": topic = "Artificial Intelligence" result = run_knowledge_graph_pipeline(topic) visualize_graph(result["graph"]) Finally, this block serves as the script’s entry point. When executed directly, it triggers the knowledge graph pipeline for the topic “Artificial Intelligence,” runs through all agent stages, and finally visualizes the resulting graph using the visualize_graph() function. It provides an end-to-end demonstration of automated knowledge graph generation. Output Generated from Knowledge Graph Execution In conclusion, we have learned how to seamlessly integrate multiple specialized agents into a cohesive knowledge graph pipeline through this structured approach, leveraging LangGraph and NetworkX. This workflow automates entity and relation extraction processes and visualizes intricate relationships, offering a clear and actionable representation of gathered information. By adjusting and enhancing individual agents, such as employing more sophisticated entity recognition methods or integrating real-time data sources, this foundational framework can be scaled and customized for advanced knowledge graph construction tasks across various domains. Check out the Colab Notebook. All credit for this research goes to the researchers of this project. Also, feel free to follow us on Twitter and don’t forget to join our 90k+ ML SubReddit. Asif RazzaqWebsite |  + postsBioAsif Razzaq is the CEO of Marktechpost Media Inc.. As a visionary entrepreneur and engineer, Asif is committed to harnessing the potential of Artificial Intelligence for social good. His most recent endeavor is the launch of an Artificial Intelligence Media Platform, Marktechpost, which stands out for its in-depth coverage of machine learning and deep learning news that is both technically sound and easily understandable by a wide audience. The platform boasts of over 2 million monthly views, illustrating its popularity among audiences.Asif Razzaqhttps://www.marktechpost.com/author/6flvq/Google DeepMind Introduces AlphaEvolve: A Gemini-Powered Coding AI Agent for Algorithm Discovery and Scientific OptimizationAsif Razzaqhttps://www.marktechpost.com/author/6flvq/Rime Introduces Arcana and Rimecaster (Open Source): Practical Voice AI Tools Built on Real-World SpeechAsif Razzaqhttps://www.marktechpost.com/author/6flvq/A Step-by-Step Guide to Build a Fast Semantic Search and RAG QA Engine on Web-Scraped Data Using Together AI Embeddings, FAISS Retrieval, and LangChainAsif Razzaqhttps://www.marktechpost.com/author/6flvq/Agent-Based Debugging Gets a Cost-Effective Alternative: Salesforce AI Presents SWERank for Accurate and Scalable Software Issue Localization

The saying goes that 80% of data collected, stored and maintained by governments can be associated with geographical locations. Although never empirically proven, it illustrates the importance of location within data. Ever growing data volumes put constraints on systems that handle geospatial data. Common Big Data compute engines, originally designed to scale for textual data, need adaptation to work efficiently with geospatial data — think of geographical indexes, partitioning, and operators. Here, I present and illustrate how to utilize the Microsoft Fabric Spark compute engine, with the natively integrated ESRI GeoAnalytics engine# for geospatial big data processing and analytics. The optional GeoAnalytics capabilities within Fabric enable the processing and analytics of vector-type geospatial data, where vector-type geospatial data refers to points, lines, polygons. These capabilities include more than 150 spatial functions to create geometries, test, and select spatial relationships. As it extends Spark, the GeoAnalytics functions can be called when using Python, SQL, or Scala. These spatial operations apply automatically spatial indexing, making the Spark compute engine also efficient for this data. It can handle 10 extra common spatial data formats to load and save data spatial data, on top of the Spark natively supported data source formats. This blog post focuses on the scalable geospatial compute engines as has been introduced in my post about geospatial in the age of AI. Demonstration explained Here, I demonstrate some of these spatial capabilities by showing the data manipulation and analytics steps on a large dataset. By using several tiles covering point cloud data (a bunch of x, y, z values), an enormous dataset starts to form, while it still covers a relatively small area. The open Dutch AHN dataset, which is a national digital elevation and surface model, is currently in its fifth update cycle, and spans a period of nearly 30 years. Here, the data from the second, third, and forth acquisition is used, as these hold full national coverage (the fifth just not yet), while the first version did not include a point cloud release (only the derivative gridded version). Another Dutch open dataset, namely building data, the BAG, is used to illustrate spatial selection. The building dataset contains the footprint of the buildings as polygons. Currently, this dataset holds more than 11 million buildings. To test the spatial functions, I use only 4 AHN tiles per AHN version. Thus in this case, 12 tiles, each of 5 x 6.25 km. Totalling to more than 3.5 billion points within an area of 125 square kilometers. The chosen area covers the municipality of Loppersum, an area prone to land subsidence due to gas extraction. The steps to take include the selection of buildings within the area of Loppersum, selecting the x,y,z-points from the roofs of the buildings. Then, we bring the 3 datasets into one dataframe and do an extra analysis with it. A spatial regression to predict the expected height of a building based on its height history as well as the history of the buildings in its direct surroundings. Not necessarily the best analysis to perform on this data to come to actual predictions* but it suits merely the purpose of demonstrating the spatial processing capabilities of Fabric’s ESRI GeoAnalytics. All the below code snippets are also available as notebooks on github. Step 1: Read data Spatial data can come in many different data formats; we conform to the geoparquet data format for further processing. The BAG building data, both the footprints as well as the accompanied municipality boundaries, come in geoparquet format already. The point cloud AHN data, version 2, 3 and 4, however, comes as LAZ file formats — a compressed industry standard format for point clouds. I have not found a Spark library to read LAZ (please leave a message in case there is one), and created a txt file, separately, with the LAStools+ first. # ESRI - FABRIC reference: https://developers.arcgis.com/geoanalytics-fabric/ # Import the required modules import geoanalytics_fabric from geoanalytics_fabric.sql import functions as ST from geoanalytics_fabric import extensions # Read ahn file from OneLake # AHN lidar data source: https://viewer.ahn.nl/ ahn_csv_path = "Files/AHN lidar/AHN4_csv" lidar_df = spark.read.options(delimiter=" ").csv(ahn_csv_path) lidar_df = lidar_df.selectExpr("_c0 as X", "_c1 as Y", "_c2 Z") lidar_df.printSchema() lidar_df.show(5) lidar_df.count() The above code snippet& provides the below results: Now, with the spatial functions make_point and srid the x,y,z columns are transformed to a point geometry and set it to the specific Dutch coordinate system (SRID = 28992), see the below code snippet&: # Create point geometry from x,y,z columns and set the spatial refrence system lidar_df = lidar_df.select(ST.make_point(x="X", y="Y", z="Z").alias("rd_point")) lidar_df = lidar_df.withColumn("srid", ST.srid("rd_point")) lidar_df = lidar_df.select(ST.srid("rd_point", 28992).alias("rd_point"))\ .withColumn("srid", ST.srid("rd_point")) lidar_df.printSchema() lidar_df.show(5) Building and municipality data can be read with the extended spark.read function for geoparquet, see the code snippet&: # Read building polygon data path_building = "Files/BAG NL/BAG_pand_202504.parquet" df_buildings = spark.read.format("geoparquet").load(path_building) # Read woonplaats data (=municipality) path_woonplaats = "Files/BAG NL/BAG_woonplaats_202504.parquet" df_woonplaats = spark.read.format("geoparquet").load(path_woonplaats) # Filter the DataFrame where the "woonplaats" column contains the string "Loppersum" df_loppersum = df_woonplaats.filter(col("woonplaats").contains("Loppersum")) Step 2: Make selections In the accompanying notebooks, I read and write to geoparquet. To make sure the right data is read correctly as dataframes, see the following code snippet: # Read building polygon data path_building = "Files/BAG NL/BAG_pand_202504.parquet" df_buildings = spark.read.format("geoparquet").load(path_building) # Read woonplaats data (=municipality) path_woonplaats = "Files/BAG NL/BAG_woonplaats_202504.parquet" df_woonplaats = spark.read.format("geoparquet").load(path_woonplaats) # Filter the DataFrame where the "woonplaats" column contains the string "Loppersum" df_loppersum = df_woonplaats.filter(col("woonplaats").contains("Loppersum")) With all data in dataframes it becomes a simple step to do spatial selections. The following code snippet& shows how to select the buildings within the boundaries of the Loppersum municipality, and separately makes a selection of buildings that existed throughout the period (point cloud AHN-2 data was acquired in 2009 in this region). This resulted in 1196 buildings, out of the 2492 buildings currently. # Clip the BAG buildings to the gemeente Loppersum boundary df_buildings_roi = Clip().run(input_dataframe=df_buildings, clip_dataframe=df_loppersum) # select only buildings older then AHN data (AHN2 (Groningen) = 2009) # and with a status in use (Pand in gebruik) df_buildings_roi_select = df_buildings_roi.where((df_buildings_roi.bouwjaar<2009) & (df_buildings_roi.status=='Pand in gebruik')) The three AHN versions used (2,3 and 4), further named as T1, T2 and T3 respectively, are then clipped based on the selected building data. The AggregatePoints function can be utilized to calculate, in this case from the height (z-values) some statistics, like the mean per roof, the standard deviation and the number of z-values it is based upon; see the code snippet: # Select and aggregrate lidar points from buildings within ROI df_ahn2_result = AggregatePoints() \ .setPolygons(df_buildings_roi_select) \ .addSummaryField(summary_field="T1_z", statistic="Mean", alias="T1_z_mean") \ .addSummaryField(summary_field="T1_z", statistic="stddev", alias="T1_z_stddev") \ .run(df_ahn2) df_ahn3_result = AggregatePoints() \ .setPolygons(df_buildings_roi_select) \ .addSummaryField(summary_field="T2_z", statistic="Mean", alias="T2_z_mean") \ .addSummaryField(summary_field="T2_z", statistic="stddev", alias="T2_z_stddev") \ .run(df_ahn3) df_ahn4_result = AggregatePoints() \ .setPolygons(df_buildings_roi_select) \ .addSummaryField(summary_field="T3_z", statistic="Mean", alias="T3_z_mean") \ .addSummaryField(summary_field="T3_z", statistic="stddev", alias="T3_z_stddev") \ .run(df_ahn4) Step 3: Aggregate and Regress As the GeoAnalytics function Geographically Weighted Regression (GWR) can only work on point data, from the building polygons their centroid is extracted with the centroid function. The 3 dataframes are joined to one, see also the notebook, and it is ready to perform the GWR function. In this instance, it predicts the height for T3 (AHN4) based on local regression functions. # Import the required modules from geoanalytics_fabric.tools import GWR # Run the GWR tool to predict AHN4 (T3) height values for buildings at Loppersum resultGWR = GWR() \ .setExplanatoryVariables("T1_z_mean", "T2_z_mean") \ .setDependentVariable(dependent_variable="T3_z_mean") \ .setLocalWeightingScheme(local_weighting_scheme="Bisquare") \ .setNumNeighbors(number_of_neighbors=10) \ .runIncludeDiagnostics(dataframe=df_buildingsT123_points) The model diagnostics can be consulted for the predicted z value, in this case, the following results were generated. Note, again, that these results cannot be used for real world applications as the data and methodology might not best fit the purpose of subsidence modelling — it merely shows here Fabric GeoAnalytics functionality. R20.994AdjR20.981AICc1509Sigma20.046EDoF378 Step 4: Visualize results With the spatial function plot, results can be visualized as maps within the notebook — to be used only with the Python API in Spark. First, a visualization of all buildings within the municipality of Loppersum. # visualize Loppersum buildings df_buildings.st.plot(basemap="light", geometry="geometry", edgecolor="black", alpha=0.5) Here is a visualization of the height difference between T3 (AHN4) and T3 predicted (T3 predicted minus T3). # Vizualize difference of predicted height and actual measured height Loppersum area and buildings axes = df_loppersum.st.plot(basemap="light", edgecolor="black", figsize=(7, 7), alpha=0) axes.set(xlim=(244800, 246500), ylim=(594000, 595500)) df_buildings.st.plot(ax=axes, basemap="light", alpha=0.5, edgecolor="black") #, color='xkcd:sea blue' df_with_difference.st.plot(ax=axes, basemap="light", cmap_values="subsidence_mm_per_yr", cmap="coolwarm_r", vmin=-10, vmax=10, geometry="geometry") Summary This blog post discusses the significance of geographical data. It highlights the challenges posed by increasing data volumes on Geospatial data systems and suggests that traditional big data engines must adapt to handle geospatial data efficiently. Here, an example is presented on how to use the Microsoft Fabric Spark compute engine and its integration with the ESRI GeoAnalytics engine for effective geospatial big data processing and analytics. Opinions here are mine. Footnotes # in preview * for modelling the land subsidence with much higher accuracy and temporal frequency other approaches and data can be utilized, such as with satellite InSAR methodology (see also Bodemdalingskaart) + Lastools is used here separately, it would be fun to test the usage of Fabric User data functions (preview), or to utilize an Azure Function for this purpose. & code snippets here are set up for readability, not necessarily for efficiency. Multiple data processing steps could be chained. References GitHub repo with notebooks: delange/Fabric_GeoAnalytics Microsoft Fabric: Microsoft Fabric documentation – Microsoft Fabric | Microsoft Learn ESRI GeoAnalytics for Fabric: Overview | ArcGIS GeoAnalytics for Microsoft Fabric | ArcGIS Developers AHN: Home | AHN BAG: Over BAG – Basisregistratie Adressen en Gebouwen – Kadaster.nl zakelijk Lastools: LAStools: converting, filtering, viewing, processing, and compressing LIDAR data in LAS and LAZ format Surface and Object Motion Map: Bodemdalingskaart – The post The Geospatial Capabilities of Microsoft Fabric and ESRI GeoAnalytics, Demonstrated appeared first on Towards Data Science.

Author(s): Felix Pappe Originally published on Towards AI. Image created by the author using gpt-image-1 Introduction In the past, extracting specific information from documents or images using traditional methods could have become quickly cumbersome and frustrating, especially when the final results stray far from what you intended. The reasons for this can be diverse, ranging from overly complex document layouts to improperly formatted files, or an avalanche of visual elements, which machines struggle to interpret. However, vision-enabled language (vLMs)models have come to the rescue. Over the past months and years, these models have gained ever-greater capabilities, from rough image descriptions to detailed text extraction. Notably, the extraction of complex textual information from images has seen astonishing progress. This allows for rapid knowledge extraction from diverse document types without brittle, rule-based systems that break as soon as the document structure changes — and without the time-, data-, and cost-intensive specialized training of custom models. However, there is one flaw: vLMs, like their text-only counterparts, tend to produce verbose output around the information you actually want. Phrases such as “Of course, here is the information you requested” or “This is the extracted information about XYZ” commonly surround the essential content. You could use regular expressions together with advanced prompt engineering to constrain the vLM to output only the requested information. However, crafting the perfect prompt and matching regex for a given task is difficult and requires much trial and error. In this blog post, I’d like to introduce a simpler approach: combining the rich capabilities of vLLMs with the strict validation offered by Pydantic classes to extract exactly the desired information for your document processing pipeline. Description of post tackled issue The example in this blog post describes a situation that every job applicant has likely experienced many times. I am sure of it. After you have carefully and thoroughly created your CV, thinking about every word and maybe even every letter, you upload the file to a job portal. But after successfully uploading the file, including all the requested information, you are asked once again to fill out the same details in standard HTML forms by copying and pasting the information from your CV into the correct fields.Some companies attempt to autofill these fields based on the information extracted from your CV, but the results are often far from accurate or complete.In the following code, I combine Pixtral, LangChain, and Pydantic to provide a simple solution. The code extracts the first name, last name, phone number, email, and birthday from the CV if they exist. This helps keep the example simple and focuses on the technical aspects.The code can be easily adapted for other use cases or extended to extract all required information from a CV.So let us dive into the code. Code walkthrough Importing required libraries In the first step, the required libraries are imported, including: os, pathlib, and typing for standard Python modules providing filesystem access and type annotations base64 dontenv.env file into os.environ pydanticLLM output ChatMistralAILLM interface PIL import osimport base64from pathlib import Pathfrom typing import Optionalfrom dotenv import load_dotenvfrom pydantic import BaseModel, Fieldfrom langchain_mistralai.chat_models import ChatMistralAIfrom langchain_core.messages import HumanMessagefrom PIL import Image Loading environment variables Subsequently, the environment variables are loaded using load_dotenv(), and the MISTRAL_API_KEY is retrieved. load_dotenv()MISTRAL_API_KEY = os.getenv("MISTRAL_API_KEY")if not MISTRAL_API_KEY: raise ValueError("MISTRAL_API_KEY not set in environment") Defining the output schema with pydantic Following that, the output schema is defined using Pydantic. Pydantic is a Python library for data parsing and validation based on Python type hints. At its core, Pydantic’s BaseModel offers various useful features, such as the declaration of data types (e.g. str, int, List[str], nested models, etc.) and automatic coercion of incoming data into the required types when possible (e.g., converting "102" into 102). Moreover, it validates whether the incoming data matches the predefined schema and raises an error if it does not. Thanks to these clearly defined schemas, the data can be quickly serialized into other formats such as JSON. Likewise, Pydantic also allows the creation of document fields with metadata that tools such as LLMs can inspect and utilize. The next code block defines the structure of the expected output using Pydantic. These are the data points that the model should extract from the CV image.class BasicCV(BaseModel): first_name: Optional[str] = Field(None, description="first name") last_name: Optional[str] = Field(None, description="last name") phone: Optional[str] = Field(None, description="Telephone number") email: Optional[str] = Field(None, description="Email address") birthday: Optional[str] = Field(None, description="Date of birth (e.g., YYYY-MM-DD)") Converting images to base64 Subsequently, the first function is defined for the script. The function encode_image_to_base64() does exactly what its name suggests. It loads an image and converts it into a base64 string, which is passed into the vLM later. Moreover, an upscaling factor has been integrated. Although no additional information is gained by simply increasing the height and width, in my experience, the results tend to improve, especially in situations where the original resolution of the image is low. def encode_image_to_base64(image_path: Path, upscale_factor: float = 1.0) -> str: with Image.open(image_path) as img: if upscale_factor != 1.0: new_size = (int(img.width * upscale_factor), int(img.height * upscale_factor)) img = img.resize(new_size, Image.LANCZOS) from io import BytesIO buffer = BytesIO() img.save(buffer, format="PNG") image_bytes = buffer.getvalue() return base64.b64encode(image_bytes).decode() Processing the CV with a vision language model Now, let’s move on to the main function of this script. The process_cv() function begins by initializing the Mistral interface using a previously generated API key. This model is then wrapped using the .with_structured_output(BasicCV) function, in which the Pydantic model defined above is passed as input. If you are using a different vLM, make sure that it supports structured output, as not all vLMs do. Afterwards, the input image is converted into a base64 (b64) string, which is then transformed into a Uniform Resource Identifier (URI) by attaching a metadata string in front of the b64 string. Next, a simple system prompt is defined, which leaves room for improvement in more complex extraction tasks but works perfectly for this scenario. Finally, the URI and system prompt are combined into a LangChain HumanMessage, which is passed to the structured vLM. The model then returns the requested information in the previously defined Pydantic format. def process_cv() -> BasicCV: image_path: Path, api_key: Optional[str] = None llm = ChatMistralAI( model="pixtral-12b-latest", mistral_api_key=api_key or MISTRAL_API_KEY, ) structured_llm = llm.with_structured_output(BasicCV) image_b64 = encode_image_to_base64(image_path) data_uri = f"data:image/png;base64,{image_b64}" system_text = ( "Extract only the following fields from this CV: first name, last name, " "telephone number, email address, and birthday. Return JSON matching the schema." ) message = HumanMessage( content=[ {"type": "text", "text": system_text}, {"type": "image_url", "image_url": data_uri}, ] ) result: BasicCV = structured_llm.invoke([message]) return result Running the script This function is executed by the main, where the path is defined and the final information is printed out. if __name__ == "__main__": image_file = Path("cv-test.png") cv_data = process_cv(image_file) print(f"First Name: {cv_data.first_name}") print(f"Last Name: {cv_data.last_name}") print(f"Phone: {cv_data.phone}") print(f"Email: {cv_data.email}") print(f"Birthday: {cv_data.birthday}") Conclusion This simple Python script provides only a first impression of how powerful and flexible vLMs have become. In combination with Pydantic and with the support of the powerful LangChain framework, vLMs can be turned into a meaningful solution for many document processing workflows, such as application processing or invoice handling. What experience have you had with vision Large Language Models? Do you have other fields in mind where such a workflow might be beneficial? Source Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor. Published via Towards AI

With the amount of Data growing exponentially in the last few years, one of the biggest challenges has become finding the most optimal way to store various data flavors. Unlike in the (not so far) past, when relational databases were considered the only way to go, organizations now want to perform analysis over raw data – think of social media sentiment analysis, audio/video files, and so on – which usually couldn’t be stored in a traditional (relational) way, or storing them in a traditional way would require significant effort and time, which increase the overall time-for-analysis. Another challenge was to somehow stick with a traditional approach to have data stored in a structured way, but without the necessity to design complex and time-consuming ETL workloads to move this data into the enterprise data warehouse. Additionally, what if half of the data professionals in your organization are proficient with, let’s say, Python (data scientists, data engineers), and the other half (data engineers, data analysts) with SQL? Would you insist that “Pythonists” learn SQL? Or, vice-versa? Or, would you prefer a storage option that can play to the strengths of your entire data team? I have good news for you – something like this has already existed since 2013, and it’s called Apache Parquet! Parquet file format in a nutshell Before I show you the ins and outs of the Parquet file format, there are (at least) five main reasons why Parquet is considered a de facto standard for storing data nowadays: Data compression – by applying various encoding and compression algorithms, Parquet file provides reduced memory consumption Columnar storage – this is of paramount importance in analytic workloads, where fast data read operation is the key requirement. But, more on that later in the article… Language agnostic – as already mentioned previously, developers may use different programming languages to manipulate the data in the Parquet file Open-source format – meaning, you are not locked with a specific vendor Support for complex data types Row-store vs Column-store We’ve already mentioned that Parquet is a column-based storage format. However, to understand the benefits of using the Parquet file format, we first need to draw the line between the row-based and column-based ways of storing the data. In traditional, row-based storage, the data is stored as a sequence of rows. Something like this: Image by author Now, when we are talking about OLAP scenarios, some of the common questions that your users may ask are: How many balls did we sell? How many users from the USA bought a T-shirt? What is the total amount spent by customer Maria Adams? How many sales did we have on January 2nd? To be able to answer any of these questions, the engine must scan each and every row from the beginning to the very end! So, to answer the question: how many users from the USA bought T-shirt, the engine has to do something like this: Image by author Essentially, we just need the information from two columns: Product (T-Shirts) and Country (USA), but the engine will scan all five columns! This is not the most efficient solution – I think we can agree on that… Column store Let’s now examine how the column store works. As you may assume, the approach is 180 degrees different: Image by author In this case, each column is a separate entity – meaning, each column is physically separated from other columns! Going back to our previous business question: the engine can now scan only those columns that are needed by the query (Product and country), while skipping scanning the unnecessary columns. And, in most cases, this should improve the performance of the analytical queries. Ok, that’s nice, but the column store existed before Parquet and it still exists outside of Parquet as well. So, what is so special about the Parquet format? Parquet is a columnar format that stores the data in row groups Wait, what?! Wasn’t it complicated enough even before this? Don’t worry, it’s much easier than it sounds Let’s go back to our previous example and depict how Parquet will store this same chunk of data: Image by author Let’s stop for a moment and explain the illustration above, as this is exactly the structure of the Parquet file (some additional things were intentionally omitted, but we will come soon to explain that as well). Columns are still stored as separate units, but Parquet introduces additional structures, called Row group. Why is this additional structure super important? You’ll need to wait for an answer for a bit :). In OLAP scenarios, we are mainly concerned with two concepts: projection and predicate(s). Projection refers to a SELECT statement in SQL language – which columns are needed by the query. Back to our previous example, we need only the Product and Country columns, so the engine can skip scanning the remaining ones. Predicate(s) refer to the WHERE clause in SQL language – which rows satisfy criteria defined in the query. In our case, we are interested in T-Shirts only, so the engine can completely skip scanning Row group 2, where all the values in the Product column equal socks! Image by author Let’s quickly stop here, as I want you to realize the difference between various types of storage in terms of the work that needs to be performed by the engine: Row store – the engine needs to scan all 5 columns and all 6 rows Column store – the engine needs to scan 2 columns and all 6 rows Column store with row groups – the engine needs to scan 2 columns and 4 rows Obviously, this is an oversimplified example, with only 6 rows and 5 columns, where you will definitely not see any difference in performance between these three storage options. However, in real life, when you’re dealing with much larger amounts of data, the difference becomes more evident. Now, the fair question would be: how does Parquet “know” which row group to skip/scan? Parquet file contains metadata This means that every Parquet file contains “data about data” – information such as minimum and maximum values in a specific column within a certain row group. Furthermore, every Parquet file contains a footer, which keeps the information about the format version, schema information, column metadata, and so on. You can find more details about Parquet metadata types here. Important: In order to optimize the performance and eliminate unnecessary data structures (row groups and columns), the engine first needs to “get familiar” with the data, so it first reads the metadata. It’s not a slow operation, but it still requires a certain amount of time. Therefore, if you’re querying the data from multiple small Parquet files, query performance can degrade, because the engine will have to read metadata from each file. So, you should be better off merging multiple smaller files into one bigger file (but still not too big :)… I hear you, I hear you: Nikola, what is “small” and what is “big”? Unfortunately, there is no single “golden” number here, but for example, Microsoft Azure Synapse Analytics recommends that the individual Parquet file should be at least a few hundred MBs in size. What else is in there? Here is a simplified, high-level illustration of the Parquet file format: Image by author Can it be better than this? Yes, with data compression Ok, we’ve explained how skipping the scan of the unnecessary data structures (row groups and columns) may benefit your queries and increase the overall performance. But, it’s not only about that – remember when I told you at the very beginning that one of the main advantages of the Parquet format is the reduced memory footprint of the file? This is achieved by applying various compression algorithms. I’ve already written about various data compression types in Power BI (and the Tabular model in general) here, so maybe it’s a good idea to start by reading this article. There are two main encoding types that enable Parquet to compress the data and achieve astonishing savings in space: Dictionary encoding – Parquet creates a dictionary of the distinct values in the column, and afterward replaces “real” values with index values from the dictionary. Going back to our example, this process looks something like this: Image by author You might think: why this overhead, when product names are quite short, right? Ok, but now imagine that you store the detailed description of the product, such as: “Long arm T-Shirt with application on the neck”. And, now imagine that you have this product sold million times…Yeah, instead of having million times repeating value “Long arm…bla bla”, the Parquet will store only the Index value (integer instead of text). Run-Length-Encoding with Bit-Packing – when your data contains many repeating values, Run-Length-Encoding (RLE) algorithm may bring additional memory savings. Can it be better than THIS?! Yes, with the Delta Lake file format Ok, what the heck is now a Delta Lake format?! This is the article about Parquet, right? So, to put it in plain English: Delta Lake is nothing else but the Parquet format “on steroids”. When I say “steroids”, the main one is the versioning of Parquet files. It also stores a transaction log to enable tracking all changes applied to the Parquet file. This is also known as ACID-compliant transactions. Since it supports not only ACID transactions, but also supports time travel (rollbacks, audit trails, etc.) and DML (Data Manipulation Language) statements, such as INSERT, UPDATE and DELETE, you won’t be wrong if you think of the Delta Lake as a “data warehouse on the data lake” (who said: Lakehouse). Examining the pros and cons of the Lakehouse concept is out of the scope of this article, but if you’re curious to go deeper into this, I suggest you read this article from Databricks. Conclusion We evolve! Same as we, the data is also evolving. So, new flavors of data required new ways of storing it. The Parquet file format is one of the most efficient storage options in the current data landscape, since it provides multiple benefits – both in terms of memory consumption, by leveraging various compression algorithms, and fast query processing by enabling the engine to skip scanning unnecessary data. Thanks for reading! The post Parquet File Format – Everything You Need to Know! appeared first on Towards Data Science.

Author(s): Towards AI Editorial Team Originally published on Towards AI. Over the past two years, we’ve helped teams design and deploy real-world LLM systems — RAG pipelines, copilots that actually reduce load, PoCs that became products, and cut down hallucinations. One year ago, we decided to put everything we knew about building architectures around LLMs — the stack, the mistakes, the gotchas, the strategies — into a single guide: Building LLMs for Production — “the most comprehensive textbook to date on building LLM applications,” as Jerry Liu (Founder & CEO, LlamaIndex) put it. The response was amazing. People read it. People built with it. People shared it. But a few months in, our DMs started filling up: “Has the book been updated?” “Does it cover the latest models like o3 or Gemini 2.5?” “Can I use [X vector DB] instead?” “How do I choose the right model for my use case?” “What if I want to do [insert new approach] that isn’t in the book?” Fair. The landscape’s shifting fast. Inference got scaled. SLMs showed up. Context windows stretched. Costs dropped. Everything moved. If AI has taught us anything, it’s to think AI-first — not just to keep up, but to build in ways that scale. So instead of answering each DM (which we still do), we took a step back. And we decided to build something that answers all of it, now and as things evolve. The result? From Beginner to Advanced LLM Developer A 60+ hour, hands-on course that takes you from “I can prompt ChatGPT” → to deploying a production-grade RAG system with a real front end. But we didn’t just pack it with knowledge — we designed it to evolve with the field. Here’s what you walk away with: ✅ A repeatable pipeline that adapts with tools, not trends ✅ A deep instinct for how to think like an AI engineer ✅ Lifetime access and weekly updates as the ecosystem changes ✅ A private Slack for graduates + a 70,000+ builder community on Discord Because staying current isn’t enough — you also need confidence that what you ship today still holds tomorrow. That’s why we’re now running monthly live cohorts — so you stay sharp, supported, and up to date. The next cohort kicks off June 1 with a live welcome call with our CEO. Launch price: $249 (75 % off) — zero risk thanks to a 30-day money-back guarantee. 👉 Join the course here The results speak for themselves: “The course greatly expanded my knowledge of building and assessing RAG pipelines.” — Eoin McGrath “Best course out there to become an AI engineer. Planning to build my own startup based on the learnings.” — Abhijit L. “From zero to hero as an LLM Developer — a clear path to build LLM applications that can change your career.” — Luca Tanieli Even industry voices you know have shared support: “A resource I’ll return to again and again, no matter how fast the AI landscape shifts.” — Tina Huang, Lonely Octopus This course is for you: You know Python but haven’t taken an LLM past the notebook. If you’re frustrated by shallow tutorials and fragmented docs… If you want to build things that work, not just read about them… If you’re ready to take LLMs seriously and want a proven structure… There’s a roadmap. And it’s working. The next cohort starts June 1st. As soon as you join, you get full access to all course material — no need to wait for the live kickoff. You can start building right away. What You’ll Learn: 🧠 LLM Basics & Prompt Mastery Transformers, tokenization, and prompting that actually reduces hallucinations 🔍 Retrieval-Augmented Generation (RAG) Chunking, embedding models, re-ranking, query rewriting, eval, and feedback loops 🎨 Fine-Tuning LoRA, adapters, and domain-specific models that actually perform 🤖 Tool & API Integration Function calling, external tools, and chained agent workflows 🚀 Deployment & Cost Control Gradio, Streamlit, latency fixes, caching, logging, monitoring, cost tracking 🏆 Capstone Project & Certification Build and ship your own LLM app — get feedback, and leave with a portfolio-ready build If you’re thinking, “This sounds great, but what if it’s not for me?” — we get it. That’s why the course comes with a 30-day, no-questions-asked money-back guarantee. Try it. Dive into the material. If it doesn’t meet your expectations, we’ll refund you in full. 👉 Secure your spot for the June 1st cohort Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor. Published via Towards AI

Think tap to pay — but smarter and safer. Visa is tapping into AI to enhance services for its global network of customers, focused on fraud prevention, personalization and agentic commerce. Sarah Laszlo, senior director of Visa’s machine learning platform, joined the AI Podcast to discuss how artificial intelligence is powering the next generation of payment experiences. Visa processes hundreds of billions of transactions each year, so even small technological enhancements can have a large impact. The company prevents $40 billion in fraud annually. “There’s so much attempted fraud that, even though we’re very good at preventing it, marginal improvements save large dollar amounts,” Laszlo said. AI also powers Visa’s personalization systems, delivering smarter, more relevant offers and recommendations to cardholders. The company’s unique dataset presents a huge opportunity to improve transaction predictions — but also brings privacy challenges. “The key to addressing those challenges is building abstract representations of users — embeddings that capture preferences without exposing private data,” Laszlo said. The company is also working toward agentic commerce — where AI agents help customers with payments. For example, AI agents with access to payment credentials can handle transactions on behalf of consumers, like booking travel arrangements, when instructed. Laszlo shared best practices for enterprises adopting generative AI. She also recommended using open-source models when possible and developing strong relationships between governance and technical teams. In one success story, Visa used GPT-4 to convert legacy code to Python, saving $5 million, with just one engineer completing 50 conversion jobs in a quarter. To learn more, watch Laszlo’s GTC session, The Next Era of Payments: How Generative AI is Shaping the Future. Time Stamps 03:28 – Visa’s priorities for AI use. 07:09 – How Visa optimizes resources using virtual GPUs. 14:42 – AI factories and unified pipelines. 18:52 – Best practices for AI in financial services. You Might Also Like…  NVIDIA’s Jacob Liberman on Bringing Agentic AI to Enterprises Agentic AI enables developers to create intelligent multi-agent systems that reason, act and execute complex tasks with a degree of autonomy. Jacob Liberman, director of product management at NVIDIA, explains how agentic AI bridges the gap between powerful AI models and practical enterprise applications. Firsthand’s Jon Heller Shares How AI Agents Enhance Consumer Journeys in Retail Jon Heller, co-CEO and founder of Firsthand, discusses how the company’s Brand Agents are transforming the retail landscape by personalizing customer journeys, converting marketing interactions into valuable research data and enhancing the customer experience with hyper-personalized insights and recommendations. Telenor Builds Norway’s First AI Factory, Offering Sustainable and Sovereign Data Processing Telenor opened Norway’s first AI factory in November 2024, enabling organizations to process sensitive data securely on Norwegian soil while prioritizing environmental responsibility. Telenor’s Chief Innovation Officer and Head of the AI Factory Kaaren Hilsen discusses the AI factory’s rapid development, going from concept to reality in under a year.

Sponsored  When you purchase through links on our site, we may earn an affiliate commission. Here’s how it works. Last Chance to download Learn Computer Forensics -- 2nd edition (worth $159.99) for free Sponsored · May 14, 2025 11:00 EDT Claim your complimentary eBook worth $159.99 for free today, before the offer ends today, May 14! Computer Forensics, being a broad topic, involves a variety of skills which will involve seizing electronic evidence, acquiring data from electronic evidence, data analysis, and finally developing a forensic report. This book will help you to build up the skills you need to work in a highly technical environment. This book's ideal goal is to get you up and running with forensics tools and techniques to successfully investigate crime and corporate misconduct. You will discover ways to collect personal information about an individual from online sources. You will also learn how criminal investigations are performed online while preserving data such as e-mails, images, and videos that may be important to a case. You will further explore networking and understand Network Topologies, IP Addressing, and Network Devices. Finally, you will how to write a proper forensic report, the most exciting portion of the forensic exam process. By the end of this book, you will have developed a clear understanding of how to acquire, analyze, and present digital evidence, like a proficient computer forensics investigator. Free offer expires today May 14. How to get it Please ensure you read the terms and conditions to claim this offer. Complete and verifiable information is required in order to receive this free offer. If you have previously made use of these free offers, you will not need to re-register. While supplies last! Offered by Packt, view other free resources The below offers are also available for free in exchange for your (work) email: Data Quality: Empowering Businesses with Analytics and AI ($24 Value) FREE - Expires 5/6 AI and Business Rule Engines for Excel Power Users ($159.99 Value) FREE - Expires 5/7 Learn Computer Forensics -- 2nd edition ($159.99 Value) FREE – Expires 5/14 The Ultimate Linux Newbie Guide – Featured Free content Python Notes for Professionals – Featured Free content Learn Linux in 5 Days – Featured Free content Quick Reference Guide for Cybersecurity – Featured Free content We post these because we earn commission on each lead so as not to rely solely on advertising, which many of our readers block. It all helps toward paying staff reporters, servers and hosting costs. Other ways to support Neowin The above deal not doing it for you, but still want to help? Check out the links below. Check out our partner software in the Neowin Store Buy a T-shirt at Neowin's Threadsquad Subscribe to Neowin - for $14 a year, or $28 a year for an ad-free experience Disclosure: An account at Neowin Deals is required to participate in any deals powered by our affiliate, StackCommerce. For a full description of StackCommerce's privacy guidelines, go here. Neowin benefits from shared revenue of each sale made through the branded deals site. Tags Report a problem with article Follow @NeowinFeed

Online Programmer (Python and C++)GameloftSofia Sofia City Province bg2 hours agoApplyJob DescriptionGameloft Sofia studio is looking for an Online Programmer with Python and C++ to join the team.Responsibilities:Contribute to our games’ technology stack, particularly on libraries, frameworks and solutions for C++ games, tools and servers.Coordinate with other tech teams, both in the Studio and outside, to contribute to transversal, cross-game solutions.Program in the relevant languages (C++, Python, Golang);Develop, monitor, evaluate and debug solutions as needed.QualificationsYou have:Strong Software Engineering Skills.Experience in Python Programming and Software EngineeringExperience in C++ Programming and Software EngineeringExperience in Multithreaded and Distributed SystemsWorking experience in both Unix and Windows systems.Able and willing to work and learn new technologies as the need arises.Ability to work in a team.Good communication skills in English.Nice to have:Experience working with network protocols such as TCP, UDPExperience with Backend Technologies (Cloud, K8S)Experience with REST APIsExperience with DjangoExperience with DockerExperience or Interest in Game DevelopmentExperience with GolangExperience with Cross Platform Development (Mac, Xbox, Playstation, Nintendo)Additional InformationWhy Join Gameloft?You want to work in exceptional industry and create games downloaded more than 1 billion times per year.You want to be part of a talented and supportive team of pioneers who have a passion for creating video games.You want to contribute to a vibrant and dynamic atmosphere.You want to join a global company and meet great people around the world.You will work on a permanent contract with competitive remuneration.You are looking for a fun place to work in the heart of the city.Health insurance coverage, Meal vouchersGame rooms, relaxation zone, office massage, bike standStudio location at the heart of the city with easy access to mass transportation.Company DescriptionLeader in the development and publishing of multiplatform games, Gameloft® has established itself as a pioneer in the industry, creating innovative gaming experiences for 25 years. Gameloft creates games for all digital platforms, from mobile to cross-platform titles for PC and consoles. Gameloft operates its own established franchises such as Asphalt®, Dragon Mania Legends, March of Empires, and Dungeon Hunter, and also partners with major rights holders including LEGO®, Universal, Illumination, Hasbro®, Fox Digital Entertainment, Lamborghini®, and Ferrari®. Gameloft distributes its games in over 100 countries and employs 3,000 people worldwide. Every month, 1 billion unique users can be reached by advertisers in Gameloft games with Gameloft for brands, a leading B2B offering dedicated to brands and agencies. Gameloft is a Vivendi company. Create Your Profile — Game companies can contact you with their relevant job openings. Apply