What We Learned When We Started Building with AI

Where We Left Off

In the first article of this series, I shared why I changed my mind about AI in Dynamics and why I believe the way we build enterprise software is changing. This time, I want to share what happened when we actually started building with it.

At Falcon, our AI engineering team, as well as our .NET and Web teams, are already using AI in their daily engineering work. For us in Dynamics, the challenge was different: we were still experimenting and trying to understand what an effective way of working with AI would look like in the specific reality of Dynamics 365 Finance & Operations.

We needed to test the approach against the realities of ERP engineering and understand what would actually work for us. Articles, demos and other teams’ experiences could give us a starting point, but I wanted us to answer that question through hands-on work.

I wanted us to build something.

That turned out to be the beginning of a much more interesting journey than I expected.

When the Standard Path Wasn’t Available

My initial idea was to explore Microsoft Copilot Studio and Model Context Protocol for Finance & Operations, which seemed like the obvious place to start. When I looked into what we actually needed, however, I ran into a practical problem: the capabilities we wanted to explore required newer online versions of F&O than we had available in our own environments.

I even approached one of our customers to see whether we could experiment together. They were not yet on the required version either, so the conversation moved towards discussing the possibilities with Microsoft.

At that point, I had a choice: wait for the standard path to become available, or find another way to learn.

We chose to experiment.

Our AI colleagues gave us an important head start by sharing their tools, the way they approached problems and the lessons they had already learned. For us in Dynamics, the question became how those ideas would translate to an ERP system with its own business logic, metadata, framework and constraints.

For me, taking part personally was a deliberate choice. I hadn’t written hands-on code since the AX 2012 days, as my work had moved to customers, projects and consulting. Going back to the code would let me judge what AI could do for Dynamics engineers from direct experience. If we were going to ask our teams to change how they build, I wanted to have done it first.

We Started Small

We deliberately chose a contained experiment: a module for employer-funded medical benefits, drawn from our own business context. It was real enough to expose us to the complexity of F&O, while keeping the scope small enough for us to understand what was happening at every stage.

The scenario was simple enough to explain but involved real ERP concepts. An employer provides a monthly medical budget for each plan, employees claim against that budget, eligibility comes from standard benefit enrollment, and approved transactions affect the budget ledger.

Our aim was to find out whether we could use AI to design, develop and test a working F&O solution; the medical benefits module gave us a concrete way to explore that question. We installed F&O on a virtual machine, used the USMF demo company and started building.

The First Lesson: A Green Build Proves Very Little

One of the earliest lessons came up repeatedly throughout the experiment:

A successful build does not mean that the feature works.

In one case, custom tables containing business logic were missing a required declaration block. The result was a green build with empty method bodies: the code was there and the compiler was happy, but the logic was silently doing nothing. It was the kind of defect we could stare past for hours, until a behavioural test exercised the real save path and exposed it.

In another case, properties applied to a button group caused the controls to disappear after compilation, even though the build succeeded. Elsewhere, an operation appeared to fail for every single record, until we discovered that we were reading the wrong company. These were ordinary F&O problems, but together they showed why compilation alone was insufficient and why, in ERP, the environment matters as much as the code when investigating a failure.

Sometimes the AI itself was confidently wrong, which made these checks even more important. A plausible explanation could send us in the wrong direction unless we tested it against what the system actually did.

Context Matters

The quality of the result also depended heavily on how well we grounded the AI in the real system. We learned to use the standard F&O application as our reference: existing implementations showed us the patterns to follow, actual benefit enrollment data established eligibility, and the system’s behaviour helped us understand the business process. This gave us a firmer basis for decisions than an implementation that merely sounded plausible.

That grounding became especially important as we moved beyond our own new module. Building something new gave us room to shape the design; extending an existing ERP system required us to understand the behaviour and constraints already in place.

From Greenfield to Existing ERP Behaviour

We then explored how AI could help us extend standard behaviour safely and in an upgrade-friendly way. One example involved claims that exceed the monthly medical budget, where we needed to understand how the difference could eventually be handled through payroll. Before committing to a design, we investigated what the system already supported.

The investigation identified a standard chain in F&O that could support the scenario and showed us why our first idea was not the best one. This was a valuable step beyond code generation: AI was helping us investigate the existing application and challenge our design before implementation started.

The next step was to see whether that understanding would translate into an extension that worked in the running system.

For the step where an approval needs to reserve budget, we rebuilt the process on F&O’s standard workflow framework, using a real example already in the product as our reference. That gave us configurable approvals, work items and an audit trail, without having to invent our own buttons and status flow.

We then ran the process from submission through approval: a claim was submitted, an approval work item was raised, someone approved it, and the budget was reserved. To confirm that the process had worked, we checked the actual movement in the ledger as well as the screen.

This also exposed a testing limit in our setup. Approval routing involved F&O’s ClickOnce workflow editor and runtime, which our headless tests could not drive. We therefore automated the budget calculation triggered by approval and verified the routing manually, choosing the type of evidence we could obtain for each part of the process.

Testing Became Part of the Experiment

As AI generated and changed code more quickly, we needed a reliable way to establish whether the resulting behaviour was correct. Testing became an integral part of the experiment, giving us a way to check our assumptions as the solution developed rather than leaving verification until the end.

For the employee-facing part of the solution, we created a separate Python suite containing 21 tests. Eighteen run offline against a controlled gateway that simulates F&O responses, allowing us to verify the conversation logic and the safety contract we expect the agent to follow under repeatable conditions. The other three are gated live tests that validate the integration against a real, running F&O environment.

Although we develop these tests with AI assistance, their purpose is to establish what the system actually does. A successful compilation is one piece of evidence; testing the behaviour in the running system is what tells us whether the solution works.

We Started Thinking in Roles

As the experiments grew, we also began thinking about how to make this way of working repeatable across the team. Simply giving developers tools and leaving each person to work out their own approach would produce a collection of individual habits. We needed an engineering methodology, so we started organising the work around distinct roles.

One agent can help with analysis and design, another with implementation, and another with testing the resulting behaviour. The engineer remains responsible for the solution and for deciding what should happen, with the agents contributing to clearly defined parts of the work.

The tools performing those roles will continue to change. What makes the approach repeatable is the deliberate design of the roles themselves, the checks between them and the responsibilities that remain with the engineer.

And Then We Asked Another Question

Our main experiment focused on engineering the solution, but once the module was working, we began to consider what AI could offer the people using the ERP as well.

If AI can help us build the ERP, can it also help someone interact with it?

That question led us to an employee-facing agent for the medical benefits scenario. Employees can describe what they need in natural language, while the agent uses information from the ERP to determine their plan, eligibility and available budget. The experience is conversational, with a clear boundary around who makes the business decision.

The agent does not decide whether something is covered. F&O does.

The agent interprets the employee’s request and communicates the result, while the authoritative business rules remain in the system. This was a useful extension of our original experiment: it showed us how a conversational interface could make the ERP easier to interact with while keeping business decisions in the application that owns them.

We also experimented with running the language layer locally, using that work to explore the same separation between the conversational experience and the authoritative business logic. The particular model or technology was secondary to understanding how those two parts should work together.

What We Didn’t Expect

Some of the most useful moments came when the system challenged our assumptions. Successful builds hid broken behaviour, environmental problems looked like code defects, and tests failed because we had misunderstood the standard application. Agents made incorrect assumptions too, sometimes offering explanations that sounded convincing until we checked them against the running system.

Those experiences strengthened my belief that AI needs engineering discipline around it. The difficult cases were the ones where we had little reason to question the answer until we looked for evidence.

The most dangerous answers weren’t the obviously wrong ones. They were the answers that looked right.

Better prompting can help, but we also need an engineering process that catches those mistakes before we rely on the result.

There Is Work Customers Never See

One of the biggest lessons for me is how much preparation AI-assisted development requires before an engineer asks for code. We need to understand the environment, give the AI the right context and define the expected behaviour. We also need tests, clear responsibilities and a process for distinguishing a convincing answer from a correct one.

Customers may never see this work, yet it makes the visible results possible. That is why I believe our next step has to include a common engineering approach alongside access to AI tools, so that the discipline behind a successful experiment can become part of everyday delivery.

What We Are Taking Forward

After this first phase of experimentation, we can see AI contributing across the engineering process: investigating existing ERP behaviour, challenging designs, implementing solutions and helping build the tests that verify them. Each contribution still depends on engineering judgement, especially when deciding whether the evidence is sufficient to trust the result.

To carry these lessons into the team, we need shared principles that can hold as the tools change. That became our next question: how do we create a common way for our Dynamics team to work with AI, giving developers a consistent foundation for the roles, checks and responsibilities we had started to define?

Looking Ahead

Our methods, tools and understanding of what agents can do are already evolving, just as we expected. The experiment has given us practical lessons to build on as that evolution continues.

In the next article, I will share what happened when we started turning these lessons into something more structured: an internal engineering harness designed to help us work with AI in a more consistent and repeatable way.

For now, the lesson I would keep is that working well with AI takes more than clever prompts. It takes the discipline of grounding the agent in the real system and proving the resulting behaviour.

The AI is fast. The discipline is what makes that speed useful.

Next in the series: how our experiments evolved into an engineering harness—and why a shared methodology matters if AI is going to become part of the way we build Dynamics solutions.

AI-Driven Enterprise Solutions – Part 2: What We Learned When We Started Building with AI