Your own account holds data you can see in a browser and can't reach from a script: an invoice list, a dashboard, a members-only export. A plain request gets the login page instead. Scraping pages behind a login with a session cookie solves that, because the cookie carries the sign-in you already completed.
A cookie is also as sensitive as your password. This tutorial builds a command that fetches a page with your own session cookie, keeps the cookie out of your code, your logs and the saved file, and tells you when the session is no longer accepted, which a site rarely reports as an error. Use it only with accounts you own, on sites whose terms allow automated access.
Install the packages, and put your Spidra key in the environment as SPIDRA_API_KEY:
# pip install spidra requests
import argparse
import os
import re
import secrets
import sys
import threading
from http.cookies import SimpleCookie
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from pathlib import Path
from urllib.parse import urlsplit
import requests
from spidra import (
ScrapeParams,
ScrapeUrl,
Spidra,
SpidraAuthenticationError,
SpidraError,
SpidraJobFailedError,
SpidraPaymentRequiredError,
SpidraRateLimitError,
SpidraServerError,
SpidraTimeoutError,
SpidraValidationError,
)
What a session cookie is
When you sign in, the site checks your password once and gives your browser a cookie, a small value that identifies your session. The browser sends it back with every request, and that's how the site knows it's you. A script that sends the same cookie is treated as that session.
Three properties matter here. A cookie without an expiry date is a session cookie, and the browser deletes it when the session ends, while one with Max-Age or Expires lasts until that time (MDN).
A cookie marked HttpOnly can't be read by the page's JavaScript, which is why a site's session cookie doesn't appear in document.cookie after you sign in.
And the server decides when a session ends, whatever your copy of the cookie says. Sessions should end on an explicit logout and on server-side idle and absolute timeouts (OWASP).
Without the cookie, you get the login page
To try everything offline, a tiny local site stands in for a real one. It has a login form, a gated /account page, and real server-side sessions that expire and can be signed out:
TTL = 600
SESSIONS = {} # session id -> the moment it stops being valid
CLOCK = {"now": 0} # the site's own clock, so a demo can move time
LOGIN = '<h1>Sign in</h1><form method="post"><input name="user"><input name="password"></form>'
ACCOUNT = (
'<h1>Welcome, demo</h1><a href="/logout">Logout</a><p>Plan: Team. Next invoice: 1 Dec.</p>'
)def new_session():
sid = secrets.token_hex(8)
SESSIONS[sid] = CLOCK["now"] + TTL
return sid
def sid_of(cookie_header):
morsel = SimpleCookie(cookie_header).get("sid")
return morsel.value if morsel else None
def session_ok(cookie_header):
return SESSIONS.get(sid_of(cookie_header), -1) > CLOCK["now"]
def respond(handler, status, body="", headers=()):
data = body.encode()
handler.send_response(status)
for name, value in [("Content-Length", str(len(data))), *headers]:
handler.send_header(name, value)
handler.end_headers()
handler.wfile.write(data)
class Site(BaseHTTPRequestHandler):
log_message = lambda *args: None
def do_POST(self): # the login form posts here
cookie = f"sid={new_session()}; HttpOnly; Path=/; Max-Age={TTL}"
respond(self, 302, headers=[("Set-Cookie", cookie), ("Location", "/account")])
def do_GET(self):
cookie = self.headers.get("Cookie", "")
if self.path == "/logout":
SESSIONS.pop(sid_of(cookie), None)
page = {"/account": ACCOUNT if session_ok(cookie) else None, "/login": LOGIN}.get(self.path)
respond(self, 200, page) if page else respond(self, 302, headers=[("Location", "/login")])
def serve(port=8811):
"""Start the site on a local port and return its base URL."""
server = ThreadingHTTPServer(("127.0.0.1", port), Site)
threading.Thread(target=server.serve_forever, daemon=True).start()
return f"http://127.0.0.1:{port}"
Ask for the gated page without a cookie:
base = serve()
response = requests.get(base + "/account", timeout=10)
print(response.status_code, response.url.removeprefix(base))
print("redirects:", [h.status_code for h in response.history])
print("signed in:", "/logout" in response.text)Output:
200 /login
redirects: [302]
signed in: FalseThe status is 200. The request was redirected to /login, the final page loaded fine, and nothing raised an error. A script that only checks the status code thinks it worked. The real demo site, quotes.toscrape.com, does the same in its own way. Its page loads normally, with a login link where a signed-in visitor sees a logout link:
# [Quotes to Scrape](https://quotes.toscrape.com/)
[Login](https://quotes.toscrape.com/login)Get your cookie, and keep it secret
Sign in to the site in your browser, with your own account. Open DevTools (F12), go to Application in Chrome or Edge, or Storage in Firefox, then Cookies, and pick the site. Copy the session cookie as name=value. On the demo site, you can sign in at quotes.toscrape.com/login with demo and demo.
Where the cookie lives afterwards matters more than how you copied it. Keep it out of the script, and out of the command line, because both end up in your shell history and in process listings. Keep it out of URLs, because URLs get logged. Keep it out of version control. Put it in an environment variable, and read it from there:
NO_API_KEY = 3 # missing or rejected
UNKNOWN = 1
EMPTY_PAGE = 10 # the page loaded but had nothing in it
BAD_URL = 11 # the argument is not a web address
SESSION_REJECTED = 12 # the page loaded, but not as you
NO_COOKIE = 13 # nothing in the environment variable
BAD_COOKIE = 14 # the cookie is not in a usable shape
class Failure(Exception):
def __init__(self, message, code):
super().__init__(message)
self.message, self.code = message, code
CONTROL = re.compile(r"[\x00-\x1f\x7f]")
NAME = re.compile(r"[A-Za-z_][A-Za-z0-9_]*")
def load_cookie(name="SESSION_COOKIE"):
"""Read the cookie string from an environment variable, never from the command line."""
if not NAME.fullmatch(name):
raise Failure("--cookie-env takes a variable NAME, not a cookie.", BAD_COOKIE)
value = os.environ.get(name, "").strip()
if not value:
raise Failure(f"set {name} to your session cookie, as name=value; name=value", NO_COOKIE)
if CONTROL.search(value) or "=" not in value:
raise Failure(f"{name} must be one line, like name=value; name=value", BAD_COOKIE)
return value
Failure carries a message and an exit code, so a script that calls this one can tell a missing cookie from a rejected one. load_cookie refuses a value that isn't a single name=value line, and it accepts the name of a variable, never the cookie itself, so a mistake like --cookie-env session=abc123 is refused without printing it:
os.environ.pop("SESSION_COOKIE", None)
try:
load_cookie()
except Failure as failure:
print(failure.code, "|", failure.message)Output:
13 | set SESSION_COOKIE to your session cookie, as name=value; name=valueTo set it without leaving it in your history, run read -s SESSION_COOKIE && export SESSION_COOKIE in your shell, which prompts for the value without showing it. A .env file you keep out of version control, or a secret manager, works too.
Then make sure nothing prints it. These helpers mask the cookie, and every value inside it, in any text, and strip tokens from URLs before they're shown:
MASK = "[cookie]"
def secret_parts(cookie):
"""The cookie string, and each value inside it that is long enough to be a secret."""
values = [pair.partition("=")[2].strip() for pair in cookie.split(";")]
return [part for part in [cookie] + values if len(part) >= 4]
def redact(text, cookie):
"""Replace the cookie and every value in it, longest first, so no piece is left behind."""
for part in sorted(set(secret_parts(cookie)), key=len, reverse=True):
text = text.replace(part, MASK)
return text
def display_url(url):
"""A URL that is safe to print: no password, and no query string, where tokens often sit."""
parts = urlsplit(url)
host = (parts.hostname or "") + (f":{parts.port}" if parts.port else "")
return f"{parts.scheme}://{host}{parts.path}"
cookie = "session=abc123XYZsecret; theme=darkmode"
print(redact(f"The page echoed {cookie} back", cookie))
print(redact("Only the value: abc123XYZsecret", cookie))
print(display_url("https://user:[email protected]:8443/account?token=abc#top"))Output:
The page echoed [cookie] back
Only the value: [cookie]
https://example.com:8443/account
It's worth seeing what a server actually hands a browser. Signing in to the local site from Python shows the cookie's attributes, with the value masked:
def mask(value):
"""Show how long a secret is, and nothing else about it."""
return "*" * len(value)
def describe_set_cookie(header):
"""What a server asked the browser to do with a cookie, with the value masked."""
name, morsel = next(iter(SimpleCookie(header).items()))
return {
"name": name,
"value": mask(morsel.value),
"http_only": bool(morsel["httponly"]),
"secure": bool(morsel["secure"]),
"same_site": morsel["samesite"] or None,
"max_age": int(morsel["max-age"]) if morsel["max-age"] else None,
}
login = requests.post(base + "/login", data={"user": "demo"}, allow_redirects=False, timeout=10)
header = login.headers["Set-Cookie"]
print(describe_set_cookie(header))
cookie = header.split(";")[0]Output:
{'name': 'sid', 'value': '****************', 'http_only': True, 'secure': False, 'same_site': None, 'max_age': 600}http_only is true and max_age is 600 seconds. Your browser holds this value and sends it back. The cookie variable now holds it, the way your DevTools copy would.
Send it with Spidra Scrape
Pass the cookie in the cookies field of a Scrape (the docs on logged-in pages cover it). It takes the usual name=value; name=value string, or rows pasted straight from the DevTools cookie table. The cookies are used for the job and never stored. To see that the value arrives, ask httpbin.org/cookies, a test service that returns the cookies it receives. The value below is made up:
# needs SPIDRA_API_KEY
client = Spidra(api_key=os.environ["SPIDRA_API_KEY"])
page = ScrapeUrl(url="https://httpbin.org/cookies")
params = ScrapeParams(urls=[page], cookies="session=demo-abc123; theme=dark")
print(client.scrape(params).content)
{
"cookies": {
"session": "demo-abc123",
"theme": "dark"
}
}Both cookies arrived exactly as sent.
Don't trust a 200
A session the site no longer accepts doesn't produce an error. It produces the logged-out page. Send the real demo site a cookie it can't accept, the stand-in for a session that expired or was revoked, and compare it with sending nothing:
Request to quotes.toscrape.com | Account link on the page | /logout on the page | Error |
|---|---|---|---|
No cookie |
| No | None |
A made-up cookie |
| No | None |
The two responses are identical. Signed in, that same spot reads [Logout](https://quotes.toscrape.com/logout). So the script has to look for something only a signed-in page shows. Check the Markdown for a marker, a plain string, with no AI involved. Fetch the whole page for this, because content-only extraction removes menus, where account links usually live:
def check_url(url):
parts = urlsplit(url)
if parts.scheme not in ("http", "https") or not parts.hostname:
raise Failure("the address must start with http:// or https://", BAD_URL)
def fetch_page(client, url, cookie):
"""Load the page signed in. The whole page, so the menu with the account link stays in."""
page = ScrapeUrl(url=url)
params = ScrapeParams(urls=[page], cookies=cookie, scrape_mode="default")
try:
return client.scrape(params).content
except SpidraError as error:
raise explain(error, cookie) from error
def check_signed_in(markdown, marker, url):
"""A cookie the site rejects still gets a normal page. Look for what only you would see."""
shown = display_url(url)
if not isinstance(markdown, str) or not markdown.strip():
raise Failure(f"{shown} came back empty. Nothing was saved.", EMPTY_PAGE)
if marker not in markdown:
raise Failure(
f"{shown} did not show you as signed in (no {marker!r} on the page). The session may "
"have expired or been signed out. Sign in, copy a fresh cookie and retry. "
"Nothing was saved.",
SESSION_REJECTED,
)
Here's the check on a logged-out page, with a token in the URL that must not be printed:
markdown = "# Account\n\n[Login](https://example.com/login)\n"
try:
check_signed_in(markdown, "/logout", "https://example.com/account?token=abc123")
except Failure as failure:
print(failure.code, "|", failure.message)Output:
12 | https://example.com/account did not show you as signed in (no '/logout' on the page). The session may have expired or been signed out. Sign in, copy a fresh cookie and retry. Nothing was saved.Choose a marker that only a signed-in page contains, such as a logout link, an account name or a button. A phrase that also appears on the logged-out page will pass for the wrong reason.
Expiry, revocation and logout
You now have three ways a good cookie goes bad, and the local site shows each one. First, the helper that visits a page, and the baseline: a valid cookie, then one that was changed by a single character.
def visit(path, cookie=None):
response = requests.get(base + path, headers={"Cookie": cookie or ""}, timeout=10)
return response.url.removeprefix(base), "/logout" in response.text
print("valid cookie: ", visit("/account", cookie))
print("changed cookie:", visit("/account", cookie + "x"))Output:
valid cookie: ('/account', True)
changed cookie: ('/login', False)
Second, logout. Signing out of the site ends the session on the server, so every copy of the cookie stops working, including yours:
headers = {"Cookie": cookie}
requests.get(base + "/logout", headers=headers, allow_redirects=False, timeout=10)
print("after signing out:", visit("/account", cookie))Output:
after signing out: ('/login', False)So copy the cookie from a session you won't sign out of, and don't click Logout in the browser that holds it. Third, expiry. The site's own clock decides, and no setting on your side changes it. Here the local site's clock moves past the session's 600 seconds:
login = requests.post(base + "/login", allow_redirects=False, timeout=10)
cookie = login.headers["Set-Cookie"].split(";")[0]
print("fresh session: ", visit("/account", cookie))
CLOCK["now"] += TTL
print("after 600 s: ", visit("/account", cookie))Output:
fresh session: ('/account', True)
after 600 s: ('/login', False)In all three cases the page loads and the check fails. The fix is the same each time: sign in again and copy a fresh cookie. The command doesn't try to sign in for you. That would mean keeping your password in the script, and a loop of failed sign-ins is how accounts get locked.
Report errors a script can act on
The command maps every failure to a message and an exit code, and masks the cookie wherever it could appear, including in the text of an error:
RULES = [
(
SpidraAuthenticationError,
NO_API_KEY,
"the Spidra API key was rejected. Check SPIDRA_API_KEY.",
),
(SpidraPaymentRequiredError, 4, "the account is out of credits."),
(SpidraRateLimitError, 5, "too many requests. Wait a moment and run it again."),
(SpidraTimeoutError, 6, "the page took too long to read."),
(SpidraJobFailedError, 7, None), # its own message names the page and the reason
(SpidraValidationError, 8, "the request was rejected as invalid."),
(SpidraServerError, 9, "Spidra had a problem on its side. Try again."),
]
def explain(error, cookie=""):
"""The first rule that matches wins. Whatever the message says, the cookie is masked."""
for kind, code, text in RULES:
if isinstance(error, kind):
return Failure(redact(text or str(error), cookie), code)
return Failure(redact(str(error), cookie), UNKNOWN)
def write_atomic(path, text):
"""Write to a temporary file, then rename it, so a crash never leaves half a file."""
temporary = path.with_name(path.name + ".tmp")
temporary.write_text(text, encoding="utf-8")
os.replace(temporary, path)
def run(client, url, cookie, marker, out="page.md"):
markdown = fetch_page(client, url, cookie)
check_signed_in(markdown, marker, url)
path = Path(out)
write_atomic(path, redact(markdown, cookie)) # a page can echo your cookie back
return path
run fetches, checks, and only then writes the file, with any echo of your cookie masked. A rejected session saves nothing, and an earlier file stays as it was.
The finished command
class QuietParser(argparse.ArgumentParser):
def error(self, message): # argparse would echo the bad arguments, which may be a secret
self.exit(2, "error: invalid arguments (not shown, in case one holds a secret)\n")
def build_parser():
ask = QuietParser(allow_abbrev=False, description="Save a signed-in page, using your cookie.")
ask.add_argument("url")
ask.add_argument("--marker", required=True, help="text only a signed-in page shows")
ask.add_argument("-o", "--out", default="page.md")
ask.add_argument("--cookie-env", default="SESSION_COOKIE", help="NAME of the variable")
return ask
def make_client():
if not os.environ.get("SPIDRA_API_KEY"):
raise Failure("set SPIDRA_API_KEY to your Spidra key first.", NO_API_KEY)
return Spidra(api_key=os.environ["SPIDRA_API_KEY"], api_url=os.environ.get("SPIDRA_API_URL"))
def main(argv=None, client=None):
args, cookie = build_parser().parse_args(argv), ""
try:
check_url(args.url)
cookie = load_cookie(args.cookie_env)
path = run(client or make_client(), args.url, cookie, args.marker, args.out)
except Failure as failure:
print(f"error: {redact(failure.message, cookie)}", file=sys.stderr)
return failure.code
print(f"Saved {path}")
return 0
Save the pieces as fetch_signed_in.py, set SESSION_COOKIE as above, and run python fetch_signed_in.py https://example.com/account --marker /logout -o account.md.
There's no --cookie option on purpose, and the parser doesn't echo bad arguments back, in case one holds a secret. A page that loads signed in is saved.
Anything else prints one error: line and exits with its own code: 3 for the Spidra key, 7 for a page that couldn't be read, 10 for an empty page, 12 for a session the site didn't accept, 13 for a missing cookie.
Access boundaries
A session cookie is your own signed-in access, and it opens exactly what your account can open. It doesn't give you more, and this tutorial doesn't help you reach a login you aren't allowed to use. A few rules follow:
- Your own account only. Never use a cookie from someone else's session, and never share yours.
- Check the terms. Many sites forbid automated access even to your own account. If the site has an API or an export, use that instead.
- Read, don't change. This command only reads pages. Don't point it at anything that submits a form or changes data.
- Keep what you need. Save the fields you use, not the whole account, and delete saved pages that hold other people's details.
- Sign in yourself. If your login uses a one-time code or single sign-on, you complete it in your browser, and then copy the cookie your own session produced.
Spidra's built-in readers for popular sites never send your cookies. Read our guide on whether web scraping is legal before you collect from any site.
Choosing an approach
Situation | Use |
|---|---|
The site offers an API or an export for your data | That. It's the supported way in. |
One page you can already see in your browser | Copy the session cookie and scrape with |
A simple login form, run from a script | Log in with browser actions in the same Scrape. Values you type are never sent to the AI and are masked in your logs. |
A login with a CAPTCHA, a one-time code or single sign-on | Sign in yourself, then pass the cookie. |
Unattended runs for days | A cookie expires, so plan to copy a fresh one, or use the site's API. |
What this doesn't handle
Sessions end, and this command reports that instead of fixing it. Some sites tie a session to a device or network, and may reject a copied cookie for that reason. A page whose marker changes needs a new marker. And a cookie belongs to one site, so copy it from the domain you're scraping.
From one page to your own data
You now have a command that fetches a signed-in page, keeps the cookie out of everything it writes, and says plainly when the session has gone. That last part is what separates a demonstration from something you can rely on.
Here is what Spidra adds when the job grows past one page:
- Your cookie goes into one field and nowhere else. It's used for the job and dropped. It isn't stored.
- One login, many pages. The same
cookiesfield works on Scrape, Batch scrape, Crawl and Map, so up to 50 pages go through one batch, a crawl can read a whole members area, and Map lists the URLs behind a sign-in. - A browser loads the page as you. Signed-in pages that build themselves with JavaScript work, which a plain request can't read.
- Simple logins can run in the same call. Browser actions fill the form, and what you type is never sent to the AI and is masked in your logs.
- Failures are clear and free. A page that can't be read raises an error that says why, and isn't charged.
- Data, not just pages. Ask for Markdown, or send a schema and get the fields back as Python data.
If the site has an API, use it. If your sessions expire every few hours and you need unattended runs for weeks, a cookie isn't the right key for that job. But for the data in your own account that you can see and can't script, a session cookie and a Scrape are the shortest path.
Create a free account, sign in to the demo site with demo and demo, copy its cookie, and run fetch_signed_in.py on it. Then point it at your own account's page. The docs cover the cookie formats, and our page on logged-in scraping has the overview.