no issue
- The test was failing because the match for a the data has changed, which was returned in a short format YYYY-MM-DD.
- Added a new matcher for short date format -anyShortDate. Can be used for match dynamic dates in short format
refs https://github.com/TryGhost/Toolbox/issues/214
- Having to fill out `process.env` variables in tests is a frustrating developer experience. Test environment should be configured with smart defaults, so the developer writing test cases would modify variables like this only is special test cases.
refs https://github.com/TryGhost/Toolbox/issues/214
- Calling `getMembersAPIAgent` and `getAdminAPIAgent` separately was booting Ghost twice, which caused a significant performance degradation.
- Additionally, having two calls was slightly ugly and having once utility function that delivers multiple agents at once feels like more readable syntax
- Often in our API we want to check that the location string looks roughly right for a resource
- At the moment we're matching any String, this upgrades the check to look for resource URLs
- I noticed our test fixture theme was still throwing errors during tests because it had partials that are no longer in use
- Updated the docs on how to update Casper to include ensuring Casper was on a valid version AND removing Casper before
copying it across
- Performed the exact steps in the README
- This ensures that the Casper version we use in our tests is not throwing tonnes of errors and is up to date
refs https://github.com/TryGhost/Team/issues/1338
This adds some initial scaffolding to make it easier to test Members with Stripe
- `mockStripe` method to disable the network, so we can use nock to intercept
Stripe API calls
- `setupStripe` method to be called _before_ getting an agent so that Stripe is
configured
- `getMembersAPIAgent` to get an agent for the Members frontend API
refs: https://github.com/Ghost/Ghost/commit/b5ee17b25
- When moving the site tests into e2e-api they broke because they were getting the changed settings value from the settings test
- The solution is to ensure the settingsCache is reset between tests, which is what this commit does
- This commit also renames the shutdown method to reset, because this is not a permanent operation
- It also renames the resetDb method to resetData, because the concept is we want the internal data to be reset, not just the DB
no-issue
Because we were returning the call to `boot`, rather than awaiting it,
it meant that once a test was provided with an API Agent the Ghost
application hadn't necessarily started. This was noticed whilst working
on the Members API which requires the frontend to be loaded.
When the frontend is loaded we must also wait for the url service to
have finished initialising, so that we do not run into 503 maintenance
errors when hitting the frontend.
no-issue
The Members API is served on a different endpoint to the Admin API, and
also requires that the frontend is booted. This agent is to be used when
testing the Members API, e.g. Stripe webhooks or Members config.
- The admin API test agent doesn't boot the Ghost frontend, which is where members routes are currently located
- This means we can't test members properly as we don't have access to webhooks etc
- This change adds a members option to getAdminAPIAgent(), which in turn enables the frontend
- We do it this way so that we can easily change the implementation later, e.g. if we have the option to boot members directly
- When starting ghost for e2e tests we create a content folder in the os tmp dir
- This means that the folder can change between suites as ghost is started and restarted
- For the most part this is fine, but theme storage caches the path to config (which makes sense, it's not meant to change whilst Ghost is in-memory)
- This is a quick-n-dirty fix that just makes it possible to update that path in the tests, so we know it's in sync
- Ideally we'd not cache the path, use a function to fetch it etc, or fully reset the theme storage layer, but this is the fix I have working today
and so it's going in to unblock things for now
- Can now easily mock labs flags as enabled or disabled using mockManager
- Updated some bits of code that directly mocked labs
- Aside: improved error thrown when things go wrong booting Ghost
refs: https://github.com/TryGhost/Toolbox/issues/158
- continuing to turn this into a reference test suite for how to do things with the new framework
- use fixtureManager and mockManager properly
- testUtils should _not_ be required anymore!
- TODO: cleanup labs mocking and side effect assertion using models
refs: https://github.com/TryGhost/Toolbox/issues/158
- rather than just exposing any, anything and string matching, expose more specific matchers.
- this was triggered by `any(Date)` not working for dates in our API
- it seems poor to match `any(String)` for something we want to be a well formatted date
- establishes the pattern of using our defined matchers instead of requiring any/anything from jest
refs: https://github.com/TryGhost/Toolbox/issues/158
- Working up to having these be the defacto "reference" tests
- Changing this allows for the fact that there will be a getContentAPIAgent, and probably more in future
- Abstracting away the path to a single location will make updating API paths easier later too
refs TryGhost/Toolbox#158
refs TryGhost/Toolbox#129 (comment)
We now have @tryghost/jest-snapshot - a suite of utilities for using snapshots with mocha, should, etc
The @tryghost/express-test lib also uses it to provide chained snapshot assertions
This library was created because all the existing implementations I could find, including the in use @ethanresnick/chai-jest-snapshot
didn't properly support property matching. @ethanresnick/chai-jest-snapshot supposedly supported it, but the implementation was incorrect
and frequently lead to false test passes.
This library also has (after some back and forth) path resolution so that snapshots can live in a local __snapshots__ folder
refs a3cc66be50
- in the referenced commit, I made a util to speed up resetting the DB
for SQLite by copying the database file
- I inadvertently removed an optimization we had before - where we
truncate the tables and insert the fixtures instead of dropping the
entire database
- this would be missing on MySQL tests
- this seems to have a big difference so this commit re-adds the
optimization in
The idea here is to keep the concept of _what_ we're asserting (that an email was sent)
away from the implementation of how we check that assertion
This means that tests can have a consistent api like .sentEmail(), and if we change how that function works,
we only have to update the assertion function in the MockManager
Much more cleanup to come behind the scenes, but the aim is to make the tests as clean as possible
- encapsulated concerns within individual objects
- this will allow us to refactor these into classes or move them around later
- also makes it clearer how methods like restore relate to other methods
- e.g mocks.restore() restores mocks, whilst fixtureManager.restore() restores the database
- Ensure that the DB is always reset between tests
- We assume that the DB will be torn down and rebuilt between tests
- Without this, previous tests clobber the current tests, and it's not possible to tell when running files in isolation
which makes developing really tricky
closes: https://github.com/TryGhost/Toolbox/issues/193
- Our Casper fixture was several years out of date.
- I'd already updated the ghost-api to point to the latest API version, which was the main difference
- This makes sure the full fixture is up to date and correct, and therefore that we're truly testing if Ghost right now works
- It also adds instructions for how to update it in future
GOTCHA: the mock-express-style tests are failing if the API difference between test-theme and casper are different
- I've tried to look into why this is - it's something to do with the overridden route settings not working properly if the API version changes
- Given that we may not keep this style of testing AND we are definitely not keeping API versions, I'm ignoreing this for now
- To get around it, I'm setting both themes to be v4 API, so that Casper is exactly as in main
refs: https://github.com/TryGhost/Toolbox/issues/193
- The Casper fixture in the codebase is copied to a tmp folder structure every single time we start Ghost in an e2e test
- We shouldn't keep files here that don't actually get used!
- The files I'm removing are src files, used for building, but we already have the built files
refs: https://github.com/TryGhost/Toolbox/issues/168
- Upgraded the remaining themes to be pinned to the canary API
- This required one minor fix because the edit URL has changed for authors in v4/canary
- I also ripped out everywhere that the theme ghost-api version was being pinned to canary, as this was unreliable and only happening in a few places
- It's also going to be unnecessary code as soon as we finish changing to only having one API version
refs: https://github.com/TryGhost/Toolbox/issues/168
- The fixture represented Casper at Ghost version 1.0 which pre-dated the introduction of the posts_per_page config in package.json
- When Casper was upgraded to 2.0 in the fixtures, the lack of pagination broke the e2e tests for pagination
- This change introduces proper code to stub and override posts_per_page rather than keeping and using the old casper-1.4 fixture
- It also required me to remove a handful of CSS-based checks which are no longer true in the new theme version, but also didn't really add anything to the tests
refs: https://github.com/TryGhost/Toolbox/issues/168
- Having large theme fixtures makes tests run slower, so we're working to reduce them
- This fixture was a full copy of a very old version of Casper, but all we needed was a handful of files so that the tests can run
- This theme is also used for testing members, not just price data so I've renamed it for clarity
- The remaining files is the bare minimum we need to test these features
refs: https://github.com/TryGhost/Toolbox/issues/168
- This file was pinning various tests to v3
We're going to be dropping the idea of having multiple versions of the API in each Ghost version.
Because this has not achieved the goal of making it easier to make breaking changes, but it has
created an ordinate amount of technical debt and maintenance overhead.
As we know this is going away in the next major, there is no benefit to us constantly running tests
that check if those versions still work, especially given how long they take.
Instead we're starting work to ensure that all of our test work on canary, and that canary has
excellent test coverage so that we can be sure that our one API version works really well and that
any changes, no matter how subtle are deliberate, tracked and understood.
refs https://github.com/TryGhost/Team/issues/1252
We need a way to signal whether or not a Tier is active or archived, and
we'll be using the active flag in the same way we do for Offers.
Since we now have 2 products by default for all ghost sites, free and default paid, the usage of default product which so far was using first product needs to be updated to use the first paid product.
- updates default product usage to use first paid tier
- updates tests
refs: https://github.com/TryGhost/Toolbox/issues/150
Instead of loading all themes for each set of tests in the e2e suite, only load the frontend at all for frontend tests, and only load themes for the theme tests.
refs https://github.com/TryGhost/Toolbox/issues/158
- To make the development quicker the copy-paste code was left in the codebase but gave immediate smell. Removed it to stop bothering me every time I look at the related code :)
refs https://github.com/TryGhost/Toolbox/issues/129
- The "Origin" header is require in Admin API requests so it makes sense to bake it into a default request made by most tests. Reduces unnecesary fluf around test request setup and removes a "config" dependency in each tests suite using the "e2e framework"
refs https://github.com/TryGhost/Toolbox/issues/158
- Allows for much smaller amount of code to configure a test to work with chai-jest-snapshots. They now work automatically for all regression tests and could be enabled for other suites by adding the "--require=./test/utils/snapshots.js" parameter in respective test:* package script
- Regenerated snapshot for authentication test as the naming structure
changed a little with the snapshot metadata being taken on a higher
level in the test (uses the suite name instead of a specific describe it
used to be called from)
refs https://github.com/TryGhost/Toolbox/issues/158
- Moving common mocking/stubbing/spying logic into an outside utils module allows test suites to keep agnostic towards which framework powers mocking etc. Should also substitute email service stubbing used in multiple places
- Bonus, got rid of should dependency, which is deprecated
no issue
- The response from the server might not always contain an array with errors (for example when there's a generic 503 error). This changes prevents tests from hanging and adds at least some relevant output
refs https://github.com/TryGhost/Toolbox/issues/129
- Adds content folder creation based on fixtures for e2e tests using the e2e framework
- Mostly a copy-paste for now from existing e2e utils with some simplifications, so we can "complicate things" later only if needed
refs 24505db918
- The code is heavily borrowed from the referenced commit by @ErisDS. It starts a fresh abstraction layer for all e2e tests to use, provides simle interface for request agent initialization and state management (read the e2e-framework top file description for more)
- Main methods the framework exposes are:
- getAgent - to initialize request agent with a fresh Ghost instance
- initFixtures - to initializer a fresh db state
- resetDb - cleans up the db state if there were state manipulations during the test suite run (POST, PUT requests or non-default fixtures)
refs https://github.com/TryGhost/Toolbox/issues/129
- Having a fresh module should allow building new testing utility concepts from the ground up without being tied by the mystery baggage of the legacy localUtils/testUtils
- The outline of the concepts the framework will be handling is in the commont on the top of the e2e-framework file
refs https://github.com/TryGhost/Toolbox/issues/152
- Being able to set up test suites without blocking a port opens a door to new ways to run tests - for example this has been one of the blockers for running mocha tests in parallel
- Additional benefit is lighter statrup, which reducec the test execution time slightly. Doesn't seem like much but these things stack up!
no issue
- we've seen some instances of SQLite saying "database disk image is malformed"
- I think this happens because we copy the file whilst we are still connected to
the old DB
- this commit destroys the connection before copying the clean file to the live
DB file
