MACALCHEMIST OFFLINE SITEMAP MANAGER v0.1.6
================================================

Map your site without crawling your site.

A free, local-first sitemap maintenance and audit utility.

WHY THIS EXISTS
---------------
Most sitemap generators discover pages by crawling the live website. That is
convenient, but it also means generating crawler traffic against the server.
One reason Offline Sitemap Manager exists is very practical: a live-site
sitemap crawler once got the developer's IP blocked from his own website.

This program takes the opposite approach. It analyses a local copy, backup,
mirror or suitable export of the website instead.

WHAT IT DOES
------------
Offline Sitemap Manager maintains an approved sitemap baseline and helps you
review changes before generating replacement files. It can also surface useful
site-maintenance problems rather than merely collecting URLs.

It can flag or help review:

* ordinary new sitemap candidates
* canonical URLs which point somewhere else
* noindex pages
* approved URLs whose local files are missing
* unresolved internal links discovered from approved pages
* changed approved pages where lastmod may need attention
* historical/local pages which are not represented in the approved sitemap

The workflow is intentionally conservative: inspect first, approve deliberately,
then generate proposed output into a separate folder.

SAFE / USEFUL / FREE
--------------------
* No live-site crawling by the sitemap engine.
* No FTP, SSH, hosting-panel or API credentials.
* No automatic changes to the live website.
* No subscription.
* No scan quota.
* No artificial program URL cap. Standard sitemap-protocol limits still apply.
* Free software under GNU GPL v3.
* Coffee donations are entirely optional.

The Help menu contains optional browser/mail links for support and contact.
Those links open only when you deliberately choose them; the sitemap engine
itself remains local-only.

IMPORTANT LIMITATION
--------------------
You need a local copy, backup, mirror or suitable export of the website.

For traditional HTML/PHP/file-based sites this is often exactly what you already
keep as a backup or working copy. For database-driven CMSs, hosted site builders
or JavaScript applications, the filesystem alone may not represent every public
URL or its final metadata. Such platforms may require a separate helper/exporter.

Any future platform-specific helper should remain separate from the stable core
program. A helper can prepare a local snapshot or manifest for this program
without turning the sitemap manager itself into a collection of fragile CMS
integrations.

REQUIREMENTS
------------
Python 3.9 or newer is required (the XML writer uses ElementTree.indent).
Tkinter and a graphical desktop/display are required for the GUI.

Verified runtime results:
* Python 3.9: all 32 regressions and the self-test passed in isolated Docker
  (maintainer-reported verification). Python 3.9 GUI startup remains untested.
* Python 3.10.12: all 32 regressions, self-test and Tkinter GUI startup passed
  on Linux Mint 21.
* Python 3.13.7: all 32 regressions and self-test passed without a GUI. This
  Homebrew installation lacks Tkinter; its GUI was not tested.

These results apply to the tested interpreter installations; GUI use requires
Tkinter support in the interpreter selected to launch the application.

On Debian/Ubuntu-family Linux systems Tkinter may be provided separately as:

    sudo apt install python3-tk

TKINTER TROUBLESHOOTING
The Python interpreter you select must include Tkinter support. An error such
as "No module named '_tkinter'" can indicate that this Python installation lacks
it, rather than an application defect. If another Python installation (for
example Homebrew Python) lacks Tkinter, try your distribution's Python:

    /usr/bin/python3 offline_sitemap_manager_v016.py

The v0.1.6 GUI was successfully launched this way on Linux Mint.

No third-party Python packages are required. Neither pip nor PyInstaller is
needed to run this source release. The self-test and regression suite do not
need a graphical display.

RELEASE SCOPE — v0.1.6
----------------------
This is a Linux/Python source release incorporating two maintenance patches.
The v0.1.6 GUI was manually launched successfully on Linux Mint using
/usr/bin/python3. The renamed source also passes automated regressions and
self-test. Windows executable packaging is deferred.

Maintenance fixes:
* Existing PDF/image/ZIP links no longer crash scans; missing page targets still
  produce warnings, and ordinary page discovery remains available.
* Sitemap filenames are validated consistently on project loading, import,
  editing and generation. Duplicate/case-conflicting names, sitemap_index.xml
  and Windows-invalid/reserved names are rejected before output creation.
* New filesystem-derived URLs percent-encode spaces, Unicode and literal
  percent, hash and question-mark characters. Existing approvals are not rewritten.
* Local mapping rejects traversal, foreign-host redirects, malformed URLs and
  symlinks escaping the selected website root. Internal aliases, slashless
  directories, directory indexes and relative-link resolution remain supported.
* DATAFILE traversal and external symlink matches are excluded. Unsafe/no-match
  patterns retain stored lastmod values and produce diagnostics.
* Scans, copied diagnostics and output reports explain blocked mappings,
  unresolved aliases and supplementary error-page omissions.

STARTING THE PROGRAM
--------------------
Extract the source ZIP, open a terminal in its extracted top-level folder, then
run:

    python3 offline_sitemap_manager_v016.py

The launcher offers:

    Open existing project...
    Create new site project...

The Help menu is available from both the launcher and the main program.

QUICK START — BUNDLED DEMO
--------------------------
For repeatable testing, first copy the extracted release folder to a temporary
or separate writable location. Opening/scanning a project updates its state;
the shipped demo starts with five approvals and no scan history.

1. Run offline_sitemap_manager_v016.py.

2. Click "Open existing project...".

3. Open:

       demo_project/sitemap-project.json

4. The project uses the bundled local dummy site:

       dummy_site/

   Its public URL is deliberately fictional:

       https://example.test/

5. Click "Initialise current snapshot" once.

6. Click "Audit existing path..." and enter:

       /

   The audit is deliberately designed to find:

   * articles/second.html        — ordinary CANDIDATE
   * orphan.html                 — ordinary unlinked CANDIDATE
   * private/noindex.html        — NOINDEX
   * legacy.html                 — CANONICAL_OTHER

7. Approve articles/second.html into "articles".
   Approve orphan.html into "main".

8. Leave the NOINDEX and CANONICAL_OTHER examples unapproved.

9. Generate the proposed files.

   Expected approved totals:

       main:      4
       articles:  3
       total:     7

PROJECT CREATION
----------------
The New Site Project form asks for:

* site name
* base URL, for example https://example.com/
* project folder
* local website root
* optional existing XML sitemap file(s)

If no sitemap is imported, the project starts with an empty approved baseline
and one sitemap group:

    main -> sitemap.xml

You can add further sitemap groups later.

NORMAL WORKFLOW
---------------
1. INITIALISE CURRENT SNAPSHOT

   Do this once when setting up a project. It records the current local site so
   routine scans focus on genuine changes rather than presenting every existing
   historical file as new.

2. SCAN FOR CHANGES

   Use this for routine maintenance after files have been added, changed or
   removed from the local website copy.

3. AUDIT EXISTING PATH...

   Use this deliberately when you know an older section contains sitemap gaps.
   For example:

       /
       /articles/
       /archive/2024/

4. REVIEW

   Nothing found by the scanner/auditor is automatically approved.

5. GENERATE PROPOSED FILES...

   Output is written to a separate timestamped directory. Review it before
   uploading anything to the website.

REVIEW STATUSES
---------------
CANDIDATE
    A locally resolvable page which can be considered for the sitemap.

NOINDEX
    The page declares noindex. Normally do not put it in the XML sitemap.

CANONICAL_OTHER
    The page declares a canonical URL different from itself. This is deliberately
    surfaced because it may be correct, or it may reveal a broken/stale canonical.

LINK_UNRESOLVED
    A discovered local-site link does not resolve to a page file.

APPROVED_CHANGED
    A previously approved local page has changed.

APPROVED_MISSING
    An approved URL no longer resolves locally. This is a warning only. Approved
    URLs are never automatically removed.

APPROVED BASELINE
-----------------
The Approved baseline tab is intentionally append-first.

You can:
* add an approved URL directly
* add sitemap groups
* export the approved URL list

Automatic removal is deliberately not provided.

EXCLUSIONS
----------
The Exclusions tab contains URL-path prefixes or globs which should never be
proposed as sitemap candidates.

Generic defaults exclude only common development trees such as:

    /.git/
    /.svn/
    /node_modules/

Add site-specific archives, admin areas or development folders yourself.

LASTMOD POLICIES
----------------
Approved URLs can use:

PRESERVE
    Keep the stored lastmod unchanged.

DAILY
    Use today's date when generated.

WEEKLY
    Update at most weekly.

FILE
    Derive lastmod from the mapped local page file.

DATAFILE
    Derive lastmod from matching local data files.

MANUAL
    Use a manually touched date.

OMIT
    Do not emit lastmod for that URL.

PROJECT SETTINGS
----------------
The Project settings tab lets you review or edit:

* site name
* local website root
* default sitemap group
* sitemap group labels
* sitemap output filenames
* optional custom ErrorDocument listing

Once an approved baseline exists, changing the base URL is deliberately
restricted. Create a new project for another site instead.

CUSTOM ERROR PAGES
------------------
Custom error pages are OFF by default.

If enabled in Project settings, the program reads local .htaccess ErrorDocument
directives and lists only targets which actually exist locally.

These links appear only in the human-readable sitemap. They never enter the XML
sitemap.

PHP, JAVASCRIPT AND DYNAMIC SITES
--------------------------------
Offline Sitemap Manager reads local source files; it does not run a web server,
PHP interpreter, JavaScript engine or database.

Consequences:

* PHP-generated descriptions may occasionally expose raw template expressions.
* JavaScript-created navigation may not be discoverable as ordinary links.
* Database-generated URLs cannot be inferred unless represented in the local
  snapshot/export or explicitly added.

This is deliberate. The program is a conservative local sitemap manager and
auditor, not a browser or live-site crawler.

GENERATED OUTPUT
----------------
Depending on the project configuration, generation produces:

* one XML sitemap per configured sitemap group
* sitemap_index.xml
* sitemap.htm — human-readable approved-site index
* sitemap-change-report.txt, including mapping warnings
* sitemap-state-snapshot.json (approved rows/settings, not a full project backup)

All output is written to a new timestamped directory. XML checks cover parsing,
duplicate approved URLs and configured filenames; they are not full sitemap
schema/protocol validation, automatic splitting or a live-server check.

PROJECT BACKUPS AND LOCAL MAPPING LIMITS
---------------------------------------
Keep both sitemap-project.json and sitemap-state.json when backing up a project.
They must be writable. State may migrate automatically when opened by a newer
application version. New projects and GUI root changes normally store an absolute
website-root path; relative roots such as the bundled ../dummy_site are supported.
After moving a real project, check/reselect its root. Avoid simultaneous writers
or filesystem changes during scans; there is no project lock and path checks are
not protection against concurrent symlink replacement.

Only .html, .htm, .php and .shtml files and their directory-index forms are
page candidates. Base URLs must describe a whole site root, not a hosted
subdirectory. Query-string routes are not locally resolved. Redirect support is
a small subset of root .htaccess Redirect/RedirectMatch directives, not an Apache
rewrite engine. Encoded path separators, parent traversal, controls, malformed
percent escapes and drive/backslash forms are rejected. An in-root symlink may
be mapped, but recursive page discovery does not promise to follow linked trees.
Approvals with blocked mappings are retained; review warnings before upload.
Boundary/lastmod fallbacks do not silently remove approvals or invent file dates.

SELF TEST
---------
Run:

    python3 offline_sitemap_manager_v016.py --self-test

Run the included regression suite from the extracted release folder:

    python3 -B -m unittest discover -s tests -v

Tests write only temporary fixtures. Some symlink cases may be skipped on
systems which do not permit creating symlinks.

You can also skip the launcher:

    python3 offline_sitemap_manager_v016.py --project /path/to/sitemap-project.json

LICENCE
-------
Copyright © 2026 Stephen MacDonald-Brown.

Offline Sitemap Manager is free software licensed under the GNU General Public
License version 3 (GPLv3). The complete licence is supplied in LICENSE.txt.

GPLv3 permits commercial use as long as its terms are followed. Alternative
proprietary/commercial licensing for organisations that wish to redistribute or
integrate the software under different terms may be available from the copyright
holder.

CONTACT / BUG REPORTS
---------------------
sitemapper@macalchemist.net


WINDOWS STATUS
--------------
The existing v0.1.5 Windows executable and its build files are unchanged.
The retained build_windows_v015.bat and OfflineSitemapManager.spec still target
historical offline_sitemap_manager_v015.py; they are not a v0.1.6 build workflow
and are intentionally excluded from this Linux/Python source ZIP.
Windows rebuilds, runtime validation and distribution updates are deferred.
Do not assume the old Windows download includes these maintenance fixes.

NOTE ABOUT WINDOWS FILESYSTEMS
-----------------------------
Windows filesystems are normally case-insensitive, while many web servers are
case-sensitive. A site containing two distinct files such as Fred.html and
fred.html cannot be represented faithfully in an ordinary Windows snapshot.
This is a filesystem limitation rather than a sitemap-manager limitation.


SUPPORT
-------
If you find the program useful, you can buy me a coffee at:

    https://buymeacoffee.com/macalchemip

Coffee is appreciated, never required.

DEVELOPMENT DIRECTION
---------------------
The core sitemap engine is intentionally kept conservative and stable.

Potential future additions include:
* visual / graphical site-map HTML output
* separate platform-specific snapshot/manifest helpers

Intentionally not planned for the core program:
* live-site crawling
* FTP/SFTP/server uploads
* automatic deletion of approved sitemap URLs
