Skip to main content

Scrape modes

After an account is validated, Scrape mode is on the Scraping page and on the page that scrapes one account. The choices are Account, Bonus points, Applications, Licenses, and All. Staff also see Save HTML. A person who is not staff cannot keep the HTML. While an account is still Pending, those choices stay hidden and Validate runs every section, including Save HTML.

Choices​

The choices are Account, Bonus points, Applications, Licenses, Save HTML (staff), and All.

  • Check one or more sections. The scrape does those sections.
  • All means every section you are allowed to run. Checking it clears the others and hides them. Unchecking it shows them again.
  • Checking any other section clears All and hides it. Unchecking the last one shows All again.

What each section does​

  • Account signs in, reads Manage and the change-address form, and saves the profile. That is what moves the creds pill off Pending.
  • Bonus points signs in and saves the bonus-point table. If the numbers match the latest saved set, that set is confirmed again. If they differ, a new set is saved.
  • Applications signs in, opens each draw, and saves the draw list. A draw that matches what is already stored is left as it is. A draw whose result or hunt changed is updated, and a result change is marked on the Applications page. A draw missing from the new list is removed, with a note, only when the whole list read cleanly. Otherwise it is kept, the job says Partial, and that section can be retried.
  • Licenses signs in and replaces the license table. The Valid row is the current license. An empty or partly unreadable licenses page never removes saved licenses. The job says Partial, and that section can be retried.
  • Save HTML is staff-only. It signs in and keeps a copy of the pages it opened, including Manage, the change-address form, and licenses. By itself it does not save bonus points or draws. The queue names the folder that was kept.
  • All runs every section you can choose. For staff that includes Save HTML. The first Validate run also keeps HTML.

A scrape that fails, or that finishes Partial, keeps the pages it managed to open. A bonus-points success that did not ask to save HTML deletes those temporary pages. Staff can delete one kept copy from the Scraping page. Deleting an account deletes the copies that belong to it.

A super admin chooses how long those copies stay on Site. Failure copies and saved copies can each stay until someone deletes them, or be kept for a number of days. The page explains that the site does not delete copies on a schedule. The number is how old a copy must be before the cleanup command may remove it, and only when someone runs that command. Copies from a scrape that is still running are not part of that setting. An admin who is not a super admin does not see those choices. Until a number of days is saved, copies stay until someone deletes them.

Saved copies are listed on the Scraping page for staff. Use them from Checks.