Module 04 · The portal and its catalog · 45 min · You are: Platform team

Catalog at scale: let it find the repos

One URL per repo works for three repos, not three hundred. Teams forget to register; the platform team becomes a ticket queue. A discovery provider scans an organisation on a schedule and reads every catalog-info.yaml it finds, so "add the file" is the whole onboarding. Rules keep that open door from letting any repo invent teams or users.

Open this module in the interactive course to track progress.

Concepts

Entity provider

Backend code that pushes entities into the catalog on its own schedule (GitHub discovery, an org import). Locations are read by the catalog; providers write to it.

Rules

catalog.rules says which kinds any source may add. A location can widen that with its own rules. Anything else is refused with a warning in the log.

Orphan

An entity whose source no longer mentions it. The default strategy is delete: a cleanup every 30 seconds removes it.

Processing loop

The catalog re-reads every entity every few minutes. Schedule entity refresh (or the refresh API) puts one at the front of the queue.

Files to know

Open them in your editor or on GitHub; the bullets say what to look at.

• portal/app-config.yaml (reference, your repo): The rules and the discovery provider.
  1. Team repos may add these kinds, nothing else L209–212
  2. Scan every repo of the account for /catalog-info.yaml on main L215–224
• portal/packages/backend/src/index.ts (reference, your repo): The backend: one line per plugin or module.
  1. The discovery module, added as a backend module of the catalog L42
• portal/catalog-info.yaml (reference, your repo): The portal describes itself, like any other component.
  1. Owned by the platform team, part of the developer-portal system L9–13

Steps

1. Install the GitHub discovery module

Backend features are npm packages plus one backend.add line. The module plugs into the catalog plugin; it is not a plugin of its own.

cd ~/work/portal && yarn --cwd packages/backend add @backstage/plugin-catalog-backend-module-github@^0.14.1

Add to ~/work/portal/packages/backend/src/index.ts after backend.add(import('@backstage/plugin-catalog-backend-module-logs'));

backend.add(import('@backstage/plugin-catalog-backend-module-github'));

2. Configure the provider and tighten the rules

The provider id teamRepos is yours to choose; it names the scheduled task. Discovery replaces the hand-made checkout location, so remove that.

Change in ~/work/portal/app-config.yaml
Under catalog:.

-   rules:
-     - allow: [Component, System, API, Resource, Location]
-   locations:
+   # What any location may add. Teams own components; only the platform's
+   # own files may add Groups, Users, Domains and Templates.
+   rules:
+     - allow: [Component, System, API, Resource, Location]
+   providers:
+     github:
+       teamRepos:
+         organization: YOUR_GITHUB_USER
+         catalogPath: /catalog-info.yaml
+         filters:
+           branch: main
+           repository: '.*'
+         schedule:
+           frequency: { minutes: 30 }
+           timeout: { minutes: 3 }
+           initialDelay: { seconds: 15 }
+   locations:

Delete from ~/work/portal/app-config.yaml

-     # team-checkout's repo, registered by URL.
-     - type: url
-       target: https://github.com/YOUR_GITHUB_USER/checkout/blob/main/catalog-info.yaml

3. Describe the portal itself

create-app wrote a catalog-info.yaml owned by [email protected]. Discovery will find it, so make it true first.

cp ~/work/backstage-workshop/portal/catalog-info.yaml ~/work/portal/catalog-info.yaml

perl -pi -e 's/YOUR_GITHUB_[U]SER/YOUR_GITHUB_USER/g' ~/work/portal/catalog-info.yaml

4. Ship it

Discovery reads main, so the portal's own file only counts once it is merged.

cd ~/work/portal && git status --short

Expect app-config.yaml, catalog-info.yaml, the backend package.json and index.ts, and yarn.lock.

Propose it as a pull request. Put it on its own branch and push it:

git switch -c catalog-discovery && git add -A && git commit -qm "Catalog: GitHub discovery and rules"
git push -qu origin HEAD

Open the pull request. It prints a link:

gh pr create --fill

Open that link in your browser. Files changed shows exactly your diff.

Merge it on GitHub (Squash and merge, then Delete branch), or from the terminal:

gh pr merge --squash --delete-branch

Either way, bring your copy up to date:

git switch -q main && git pull -q

5. Restart and watch the scan

A new backend module needs a restart. Stop the portal (Ctrl+C) and start it again.

cd ~/work/portal && set -a && . ./.env && set +a

yarn start

About 15 seconds after start, the backend log shows Read N GitHub repositories (M matching the pattern). N is every repo the token can see under YOUR_GITHUB_USER; the ones without catalog-info.yaml on main add nothing.

6. Trigger a scan yourself

The schedule says every 30 minutes. In your second terminal (with .env loaded), run the task now through the catalog's scheduler API.

curl -s -H "Authorization: Bearer $PORTAL_API_TOKEN" -X POST localhost:7007/api/catalog/.backstage/scheduler/v1/tasks/github-provider:teamRepos:refresh/trigger

Give it half a minute. The old URL location is now an orphan and is deleted; discovery adds checkout back. Expect checkout and portal:

curl -s -H "Authorization: Bearer $PORTAL_API_TOKEN" "localhost:7007/api/catalog/entities?filter=kind=component" | jq -r '.[].metadata.name'

Where checkout now comes from. Expect a url: pointing at YOUR_GITHUB_USER/checkout, set by the provider:

curl -s -H "Authorization: Bearer $PORTAL_API_TOKEN" localhost:7007/api/catalog/entities/by-name/component/default/checkout | jq -r '.metadata.annotations["backstage.io/managed-by-origin-location"]'

Simplified for the course

You scan a personal account with repository: '.*'. In a company, scan the organisation with a GitHub App instead of a personal token, filter by topic or name, and import teams and users with the GitHub org provider so org.yaml is not typed by hand.

Break it on purpose: A team adds its own admins

team-checkout wants an admin group and, helpfully, defines it themselves in their repo. The push works and the portal keeps running.

You practise: finding an entity that silently never arrived, and reading the catalog's warnings in the backend log.

1. Break it

Add to ~/work/checkout/catalog-info.yaml at the end of the file

---
apiVersion: backstage.io/v1alpha1
kind: Group
metadata:
  name: checkout-admins
spec:
  type: team
  parent: team-checkout
  children: []

cd ~/work/checkout && git commit -qam "checkout: add our admin group" && git push -q

curl -s -H "Authorization: Bearer $PORTAL_API_TOKEN" -X POST localhost:7007/api/catalog/.backstage/scheduler/v1/tasks/github-provider:teamRepos:refresh/trigger

2. What you see

Expected: a checkout-admins group under Team Checkout. You get: no such group in the catalog, and checkout itself looks fine. Nothing on the entity page says why.

3. Fix it yourself

Find out why the group never arrived, and put things right the way the platform intends: team repos do not define teams.

- ---
- apiVersion: backstage.io/v1alpha1
- kind: Group
- metadata:
-   name: checkout-admins
- spec:
-   type: team
-   parent: team-checkout
-   children: []

cd ~/work/checkout && git commit -qam "checkout: groups belong in org.yaml" && git push -q

Quiz

1. What is the difference between a location and an entity provider?
   • The catalog reads a location; a provider runs on its own schedule and writes entities into the catalog
   • A location is for YAML and a provider is for JSON
   • Providers only work in production, locations only on a laptop
2. You remove a location from app-config.yaml. What happens to its entities by default?
   • They stay forever, marked as stale
   • They become orphans and are deleted by a cleanup that runs every 30 seconds
   • The backend refuses to start until you delete them by hand
3. Why can a discovered team repo not add a User?
   • Users can only be added through the UI
   • GitHub does not allow the file to contain users
   • catalog.rules lists the kinds any source may add, and User is not one of them

Field note

The day a company switches from "register your repo" to discovery, the catalog stops being the platform team's chore. It also shows how many repos have no owner at all, which is the more useful discovery.

Report a problem with this module