Author(s): Igor Novikov Originally published on Towards AI. Image by the author Crew AI is one of the most popular agentic AI frameworks out there. I’ve reviewed Autogen by Microsoft before, and it turned out to be quite buggy. This one is more mature, better documentation and generally much more stable. Architecturally they are all kinda the same though. The basic idea is a tree of execution where you have agents as nodes of the tree that can have one or both ways connections to other agents. That allows them to build various types of flows with them to solve different tasks. Let’s look at Crew basic structure: So we have: Agents: The overworked interns of your AI crew. Each one is programmed with a specific role (like “Cat Meme Analyst” or “Code Reviewer”) and a set of tools. Pro tip: Set verbose=True to hear them complain about their workload in real time. Tasks: The to-do list you slap onto your agents. “Generate 10 puns about quantum computing” or “Debug this code while pretending you’re a pirate.” Tasks can be stacked and prioritized. Processes: The “how” of chaos coordination. Crew AI offers options like Hierarchical (CEO agent barks orders at manager agents, who panic-delegate to intern agents) or Sequential (tasks move like a conga line at a retirement home). Choose wisely, unless you want your “Generate cat memes” task stuck behind “Solve climate change.” Tools: The Swiss Army knives your agent. Need to call an API, scrape the web, or send a passive-aggressive Slack message? Tools let your agents do it. Let’s build something like a marketing research system. You tell it the name of the company and it will find out all about it and its decision makers. We are going to use Google Collab for easy sharing. The Crew AI framework enables the creation of autonomous agent teams to tackle complex workflows. In this marketing research system, two crews collaborate hierarchically to generate a comprehensive company report. First let’s ask DeepSeek to create a plan for such a system: Environment Setup Install necessary packages: crewai, openai, langchain, crewai-tools, gdown, and selenium. Configure Selenium with headless Chromium (including cookie management and user-agent spoofing). Tool and Utility Preparation Set up web scraping and search tools (e.g., SerperDevTool, WebsiteSearchTool, ScrapeWebsiteTool). Create utilities for human-like delays, caching scraped HTML, and parsing HTML with an LLM. Agent Development Company Analyst: Gather general company data (industry, specialization, recent events) using web search and scraping tools. Case Studies Analyst: Retrieve relevant case studies from an internal repository (Google Doc) using semantic search. Decision Maker Analyst: Extract key executive profiles from LinkedIn using a Selenium-driven scraper. Manager Agent (Sales Strategist): Delegate tasks to the analysts, consolidate their findings, and produce a cohesive markdown report. Task Definition Define tasks with clear instructions (e.g., “Research the company ‘X’” and “Identify decision-makers”). Set expected output format (markdown report without code blocks). Crew Orchestration Organize agents into a hierarchical crew using Crew AI. Assign tasks to appropriate agents and set the process flow (manager oversees delegation). Initiate the crew with company input and execute the workflow. Integration and Testing Integrate LLM-based HTML extraction to structure raw data. Implement caching to reduce redundant scraping. Test the system against real-world data (e.g., LinkedIn pages) and adjust for dynamic content loading. Finalization Consolidate the outputs from all agents into a final report. Validate data accuracy and adjust error handling or delays as needed. Deploy or extend the system for further marketing research tasks. Now that we have a plan, let’s write the code. web_search_tool = SerperDevTool()website_scrape_tool = ScrapeWebsiteTool()case_studies_tool = GetOurCompanyCaseStudies() inputs = { 'company': "Microsoft"} company_researcher = Agent( role="Company Analyst", goal="Analyze the company, gather data by reviewing its social media and website, and identify pain points.", backstory=f"You are an experienced analyst with expertise in business and process automation at {OUR_COMPANY}. Your task is to gather information that the sales department will use to develop hypotheses for client outreach.", tools=[web_search_tool, website_scrape_tool], verbose=True, allow_delegation=False, llm=llm) case_studies_researcher = Agent( role="Case Studies Analyst", goal="Analyze the company, gather data by reviewing its social media and website, identify pain points, and determine the most relevant case studies we can present to the company.", backstory=f"You are an experienced analyst with expertise in business and process automation at {OUR_COMPANY}. Your task is to gather information and select relevant case studies, which the sales department will use to develop hypotheses for outreach to clients.", tools=[web_search_tool, case_studies_tool, website_scrape_tool], verbose=True, allow_delegation=True, llm=llm) decision_makers_researcher = Agent( role="Decision Maker Analyst", goal="Analyze the key figures of the company, their interests, communication style, and level of technical knowledge.", backstory=f"A specialist in analyzing people and decision-makers, skilled in identifying the motivation of business leaders. You work at {OUR_COMPANY}. Your task is to gather information (company activities, recent social media posts, hobbies, place of residence) about the executives of potential clients, which the sales department will use to develop hypotheses for outreach to clients.", tools=[website_scrape_tool, web_search_tool], verbose=True, allow_delegation=False, llm=llm) manager = Agent( role="Sales Strategist", goal="Manager of a team that researches a company and creates detailed reports based on data analysis and research findings about potential clients for sales team to use for their outreach strategy.", backstory="Efficiently manage the crew and ensure high-quality task completion. You're known for your ability to turn complex data into clear and concise reports, making it easy for others to understand and act on the information you provide.", verbose=True, allow_delegation=True, llm=llm_o1)task_research_company = Task( description=( "Research the company '{company}' and gather information about: industry, specialization, and recent events.\n" "The company name is exactly as it is spelled (including any special characters)—make sure you don’t mix it up with similarly named companies.\n" "Make sure to analyze their LinkedIn and website.\n" "Find similar cases from our Case Studies and mention them.\n" "Don’t draw any conclusions—just gather data into a report.\n" f"Today is {today}." ), expected_output="A fully fledge report with the mains topics, each with a full section of information. Formatted as markdown without '```'" ) sales_research_crew = Crew( agents=[company_researcher, decision_makers_researcher, case_studies_researcher], tasks=[task_research_company], verbose=True, memory=True, manager_agent=manager, process=Process.hierarchical) company_research_results = sales_research_crew.kickoff(inputs=inputs) print(company_research_results) The full version is here: Google Colab Edit description colab.research.google.com Let’s see how the code works. Company Analyst Crew The Company Analyst dives into the company’s online presence — analyzing its website and social media — to extract industry trends, recent events, and general insights. The Case Studies Analyst searches our database of case studies (downloaded from Google Docs!) to identify previous cases that might be relevant to the target company. The Decision Maker Analyst focuses on gathering details about the company’s key figures, their interests, and other subtle hints that could be useful for tailoring outreach. The Sales Strategist (Manager) serves as the orchestration engine that coordinates the flow of tasks, ensuring that data from each agent is well-structured and that nothing slips through the cracks. The LinkedIn-Focused Decision Makers Crew The LinkedIn Researcher uses our custom GetCompanyLinkedinPeople tool to scrape LinkedIn pages, carefully handling authentication via cookies and simulating human behavior with Selenium. The Data Checker validates and cross-references the information extracted from LinkedIn and other sources, helping to ensure that the final report is reliable. How the System Works When you call the kickoff method on the crew with an input like { ‘company’: ‘Microsoft’ }, the system follows a hierarchical process. The manager agent delegates tasks to specialized agents, each leveraging its own toolkit (from web search and website scraping to semantic search within our case studies) to complete their assignment. The result? A fully-fledged, markdown-formatted report. Our use of the Process.hierarchical mode ensures that the Sales Strategist (Manager) not only delegates but also synthesizes the outputs from each subordinate agent. This hierarchical workflow minimizes randomness in agents actions. With multiple agents processing data in parallel, ensuring the quality of information is crucial. The inclusion of a Data Checker agent in the second crew helps verify the integrity of the information. Tools CrewAI comes with a set of pre-build tools that cover all basic needs, but you can add your own as I did with LinkedIn parser. Each agent should be equipped with an adequate set of tools for the task. For example: Web Search and Scraping — Agents like the Company Analyst and Decision Maker Analyst use tools such as SerperDevTool and ScrapeWebsiteTool to navigate and extract data from the web. LinkedIn Scraping — Our custom GetCompanyLinkedinPeople tool, built on Selenium, mimics human browsing patterns to deal with LinkedIn’s dynamic content and login requirements. Text Extraction from Google Docs — The GetOurCompanyCaseStudies tool downloads and caches our case studies, allowing semantic search over a well-maintained repository. Conclusions Crew is a pretty good framework. It is stable and straightforward. It abstracts enough of boilerplate code but still gives you fine grained control. I find it quite easy to understand and learn. As for the agentic approach — if you run the code you will see that results are not perfect. As agentic logic is fuzzy, and not rigid like a computer program, it provides different results each time, sometimes wrong or hallucinations. On average though — it works pretty well, well enough to be useful in cases where exact correctness is not a requirement. Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor. Published via Towards AI