Writing Malware With ChatGPT

There are a lot of articles floating around about how ChatGPT can or can't write malware, and I tend to avoid them. But having been in this blended ML Security space for a while now, I thought I might have something useful to share. In this post I'll write a piece of vanilla Windows malware from scratch using ChatGPT. If you’re just here for the TLDR:

1. Malware generated by ChatGPT or (any other LLM) does not get a pass from security products just because it’s synthetically generated. The same goes for synthetically generated phish. It doesn't undo years of security infrastructure and capability build up.

2. ChatGPT helps users become more efficient and/or scale, but having more malware isn’t necessarily useful. However, being able to duplex mailslots and deploy new functionality in a few hours probably is rather useful.

3. You can write functional malware with ChatGPT (or CoPilot), it’s just a matter of asking the right questions. A lot of examples simply ask “write me malware”, which isn’t the right approach. It might prefer to write code in Python, but you can ask for anything (even nse scripts). Or if you want your malware in GO, just ask it to covert a function to GO, want it in an Android app? Just ask.

4. If you're going to use ChatGPT to write malware, you kind of already need to know how malware works. Domain knowledge + ML = Winning

5. We could probably apply code scanning rules to the output of LLMs, or use embeddings to classify code as it was being generated. How annoying would that be? Defender eating code that hasn’t even been compiled yet… or worse, needing to use Google because ChatGPT won’t generate a CreateRemoteThread function.

Code

Malware is Software

You wrote malware without jailbreaks? Malware is software. While I’d never claim to be a SWE, malware at it’s core is kind of simple. Sure you can layer on the complexity, but as long as you can get traffic back and forth through some intermediary, you’re most of the way there. If you want to know more about C2 or malware dev, I highly recommend Flying a False Flag by @monoxgas, and the Darkside Ops courses.

As for prompting ChatGPT the end-to-end is below. My objective was basically not to deal with any “red squigglies” in Visual Studio, or install external libraries, or really do anything other than open Visual Studio and go. My prompt strategy was basically to prompt until I had a solution that would compile amd had what I needed for the next step. For example, I ended up asking for code that would fix some of the type issues, ”Convert a std:string to LPCSTR.”, “Write a function to parse a vector into a string“, and so on.

The Agent

The agent took ~60 prompts to build, with “Write a function to parse the data below. Return the key and values for args, cmd, and task_id using regex. {"args": "", "cmd":"","task_id":99}” taking 10 prompts to get right. It kept wanting to use an external json library (#include <nlohmann/json.hpp>). Which is probably fair as JSON is perhaps a weird choice to shovel malware traffic back and forth. The code isn’t pretty I mostly copied code straight into the cpp file. Often times ChatGPT would give me an updated main function, but not include other functions. For example,

result from “Write a while loop that uses GetContentFromURL() and passes the output into ParseString() then passes the parsed string into RunTask”

Later prompt that overwrote the main loop

The Server

The server was relatively easier and took ~45 prompts to get everything I wanted. The primary issue I ran into with the server was actually just linking it back up with the generated agent code. Or at some point I arbitrarily decided that “agent_id” was better that “agent_uuid” and rather go back and lose work, or reprompt, I manually altered functions in both the agent and the server. There is really not much to say here, ChatGPT is pretty good with Python. Starting with uniform data structures for each server and agent to share and work from likely would’ve helped reduce prompt counts for both components.

Prompt Strategy

I preferred to tweak each prompt, or split requirements into multiple prompts. Once I got a function I liked or would at least complile, I moved onto the next thing. Prompts can build on themselves really nicely. For example, if ChatGPT generates a function you like, reference it in a downstream prompt, something like “Write the same function for this <dataclass>”. Here is the full list of final prompts,

Server Prompts:

1. Write a Python class that stores the following variables, task_id, timestamp, cmd, args, results, complete. Include a to_json method to convert the class to json.

2. Write a factory function for the Task class, where task_id is a uuid and the timestamp shows when the task was created.

3. Write a Python class that stores the following variables, agent_id, ext_ip, and timestamp.

4. Add functions to manage a FIFO task_queue

5. Add a update_agent function to update a new host_information dict variable.

6. Write a Flask API with a "todo" endpoint that takes an single agent_id parameter. Start Flask on port 80, debug should be false. If the agent_id is not in agents = {}, create it and add it to agents wehere agent_id is the key.

7. Add an POST "tasking" endpoint that creates tasks for an agent. Do not create a new agent.

8. In the to_do function, if the task_queue is empty return return jsonify({'task_id': 99})

9. Provide curl commands to test each endpoint with accurate inputs

10. Add an endpoint to get tasks in json from all agents

11. Add an endpoint to get tasks in json for an agent

12. Require "tasking" to have a valid token using a decorator

    1. In “todo”, if not agent.task_queue, return a new task with a task_id of 99

Agent Prompts:

1. Write an function to get content from http://localhost/todo?agent_id=new", use Win32.

2. Write a function to generate a unique id

3. Update main such GetContentFromURL("http://localhost/todo?agent_id=new") uses the output of GenerateUniqueId for the agent_id url parameter

4. Store uniqueID as a global variable

5. Write a function to gather a process list on Windows using Win32. Return the output as a string.

6. Use std::wout instead, and convert it back to std:cout.

7. Write function to start a "cmd.exe" process and return the output as a string. It should takes an "args" argument.

8. Add some error checking using GetLastError

9. Convert std::string cmd = "cmd.exe /c " + args; to a wstring

10. Write a function that parses the following string {"task_id": "d267566a-caa5-4d1a-bda9-1372c807ccd9", "cmd": "run", "args": ["dir"]}

11. Write a case statement to execute the right function based on run: ExecuteCommand, ps: GetProcessList. If task_id is 99, there is no task to complete.

12. Write a function to send the output back to localhost with the structure {"task_id": task.task_id, "results": output}. Use wininet.

13. Convert a std:string to LPCSTR.

14. Convert a std:string to LPCWSTR.

15. Write a function to parse a vector into a string.

16. Write a while loop that uses GetContentFromURL() and passes the output into ParseString() then passes the parsed string into RunTask

17. Add a sleep to the while loop

18. Write a function to parse the data below. Return the key and values for args, cmd, and task_id using regex. {"args": "", "cmd":"","task_id":99}

19. Write a check that says, if "No task to complete", continue

I’m sure you can already imagine how you might use a different set of prompts. As a baseline though, it’s not bad. While it still took me a few hours to get everything all lined up, I wrote nothing but print statements and have a basic piece of malware. Is it getting past Windows Defender or anything else? Probably not, our agent was eaten by Defender almost immediately. The agent fared relatively better on VirusTotal which I initally thought this might have something to do with “localhost” as my domain. But changing the domain to “moohax.substack.com” and it didn’t seem to matter (for whatever VT is worth)

moohax.substack.com