Continuous integration playbook

Actions

buildchecker will still allow the authors of the last few failed builds, as well as the @dev-experience team, to push to the main branch so as to make any changes necessary to restore the pipeline to a healthy state.

1. Follow the “Build has failed on the main branch” guide.
2. If the issue has been resolved, wait for buildchecker to unlock the branch or manually trigger a run (click “Run workflow”).

Actions

1. Check your build on Buildkite.
   • Find its link directly in the #buildkite-main channel.
   • 💡 Or run sg ci status in your shell, with the main branch checked out.
2. Search for the failing steps, and browse the logs (💡 run sg ci logs in your shell, with the main branch checked out) .
   • Look for a failure explanation: it can be a test that failed or a command that return a non zero exit code.
3. Check the previous builds on the main branch on Buildkite
   1. Are they failing with the same exact error?
      • Yes: see the Builds are failing in the main branch with the same error
      • No: see next point.
4. Is that a real failure or a flake?
   1. Restart that step. Maybe it will fail again, but if it doesn’t it’ll save you time.
      • 💡 You can go to 3. while it runs.
   2. See Is that a failure or a flake scenario
   3. Did restarting it fixed the problem?
      • Yes: that’s a flake. See the Spotted a flake scenario
      • No: see next point.
   4. Does the failure points to problem with the code that was shipped on that commit?
      1. Yes, and it’s a very quick fix that can get merged promptly:
         1. Write a short message on #buildkite-main and tell others that you’re fixing it.
         2. Submit the fix with another PR and get it merged as soon as possible.
      2. Yes, but it’s not easily and/or quickly fixed
         1. Revert the incriminating Pull Request.
         2. Open a GitHub issue mentioning the build and the context to explain to the team owning that test what happened.
         3. Checkout the PR branch.
         4. Rebase it so it includes the changes that broke it when merged in the main branch.
         5. Create a build using sg ci build main-dry-run in order to get the CI to run the same exact checks it does on the main branch.
      3. No, but it seems to fail in step or code from another team.
         1. Reach out a member of the team responsible for that test.
         2. go for a. or b. from the previous points.
   5. No, and there is suspicion of a flake.
      • Yes: that’s a flake. See the Spotted a flake scenario

Actions

1. Identify the error in common with the recent builds on Buildkite.
   • 💡 See How to use loki here
2. Find the build where the problem appeared for the first time.
   • 💡 Often it’s the first build that became red, but check that the error is the same to be sure.
3. Is this an external failure or an internal one?
   • 💡 External failures are about downloading a dependency like a package in a script or a in a Dockerfile. Often they’ll manifest in the form of an HTTP error.
   • 💡 If unsure, ask for help on #dev-chat.
   • Yes, it’s an external failure:
     1. See the SSH into an agent scenario
     2. Try to reproduce the faulty HTTP request so you can observe what’s the problem. Is it the same failure?
        • Yes: Do you know how to fix it? If no escalate by creating an incident (/incident on Slack).
        • No: escalate by creating an incident (/incident on Slack).
   • No, it’s an internal failure:
     1. Is it involving faulty state in the agents? (a given tool is not found where it should have been present, or have incorrect version)
        • See the SSH into an agent scenario
     2. Try to find an agent that recently successfully ran the faulty step (look for a green build on the main branch)
        1. Can you see a difference? If yes take note.
     3. Do you know how to fix it?
        • Yes: apply the fix.
        • No: Restart the agents to see if it fixes the problem. See Restarting the agents
          • Does it fix the problem? If no, escalate by creating an incident (/incident on Slack).

Actions

1. Escalate by creating an incident (/incident on Slack).
2. Get some help.
3. Downscale the agents to be able to observe exactly what’s going on.
   1. Update the autoscaler manifest to run a single agent
      1. cd sourcegraph/infrastructure
      2. Edit the manifest to set 1 on both maximum and mininimum agent count.
      3. cd buildkite/kubernetes/buildkite-autoscaler
      4. Run kubectl apply -n buildkite -f buildkite-autoscaler.Deployment.yaml
      5. Usekubectl get pods -n buildkite -w to observe the currently running agents (k9s works here too).
   2. From there, any change and build will run on a single agent, allowing you to observe the behaviour live.

Actions

1. What kind of step is failing?

• Is this an End-to-end tests?
  • 💡 E2E tests are fragile by nature, there is no way around it.
  • Take note.
• Is this a Docker image build step?
  • 💡 This should really not be happening.
  • Is the error about the Docker daemon?
    • Yes, this is a CI infrastructure flake. Ping @dev-experience-support on Slack in the #buildkite-main or #dev-experience channels.
    • No: reach out to the team owning that Docker image immediately.
• Anything else
  • Take note of the failing step and go to next point.

1. Is that flake related to the CI infrastructure?

• The CI infrastructure often involves:
  • Docker daemon not being reachable.
  • Missing tools that we use to run the steps, such as go, node, comby, …
  • Errors from asdf, which is used to manage the above tools.
• Yes: ping @dev-experience-support on Slack in the #buildkite-main or #dev-experience channels.
  • If nodoby is online to help:
    • Reach out for help in #dev-chat
    • Try Restarting the agents and restart the build.

1. Is that flake related to the code:

• See the process describe in the flaky tests page

Actions

1. Immediately restart the faulty step.
   • 💡 It will save you time while you’re looking at the logs.
   • Is the step passing now?
     • Yes: See Spotted a flake scenario
     • No: Give it another try, and see next point.
2. Check on Grafana if there are any occurrences of the failures that were previously observed:
3. Go the the “Explore” section
4. Make sure to select grafanacloud-sourcegraph-logs in the dropdown at the top of page.
5. Scope the time window to 7 Days to make sure to find previous occurrences if there are any
6. Enter a query such as {app="buildkite"} |= "your error message" where “your error message” is a string that identiy approximately the failure cause observed in the failing step.
7. Is there a build that failed exactly like this?
   • Yes:
     1. 💡 Double check that you’re looking at that the same step by inspecting the labels of message (click on the line to make them visible)
     2. Yes, that’s a flake. See the Spotted a flake scenario
   • No: it’s not a flake, reach out the team owning those tests.

Actions

1. Use kubectl get pods -n buildkite -w to observe the currently running agents (k9s works here too).
2. In a different terminal, run kubectl -n buildkite rollout restart deployment buildkite-agent.
3. Wait a bit to see the agents restarting completely.
4. Restart the faulty build and observe it the problem is fixed or not.
   • If necessary: escalate by creating an incident (/incident on Slack).

Actions

1. Find the pod you want to SSH into with one of the following methods:
   1. Use kubectl get pods -n buildkite -w to observe the currently running agents and get the pod name (k9s works here too).
   2. From a Buildkite build page, click the “Timeline” tab of a job and see the entry for “Accepted Job”. The “Host name” in the entry is also the name of the pod that the job was assigned to.
2. Use kubectl exec -n buildkite -it buildkite-agent-xxxxxxxxxx-yyyyy -- bash to open a shell on the Buildkite agent.