diff --git a/.flake8 b/.flake8 new file mode 100644 index 0000000..36e209e --- /dev/null +++ b/.flake8 @@ -0,0 +1,9 @@ +[flake8] +exclude = + __pycache__, + .venv, + .git, + dist, + build +max-line-length = 100 +ignore = E203 diff --git a/.github/workflows/push.yaml b/.github/workflows/push.yaml new file mode 100644 index 0000000..ec299df --- /dev/null +++ b/.github/workflows/push.yaml @@ -0,0 +1,38 @@ +name: Robot Framework statuschecker CI + +on: [push, pull_request] + +jobs: + build: + + runs-on: ubuntu-latest + strategy: + matrix: + python-version: [3.7, 3.9, pypy3] + rf-version: [4.0.3, 4.1.2] + + steps: + - uses: actions/checkout@v2 + - name: Set up Python ${{ matrix.python-version }} with Robot Framework ${{ matrix.rf-version }} + uses: actions/setup-python@v1 + with: + python-version: ${{ matrix.python-version }} + - name: Install dependencies on Python + if: matrix.python-version != 'pypy3' + run: | + python -m pip install --upgrade pip + pip install -r requirements-dev.txt + pip install robotframework==${{ matrix.rf-version }} + - name: Install dependencies on pypy3 + if: matrix.python-version == 'pypy3' + run: | + python -m pip install --upgrade pip + pip install robotframework==${{ matrix.rf-version }} + - name: Run lint + if: matrix.python-version != 'pypy3' + run: | + inv lint + - name: Run tests + run: | + python --version + python test/run.py diff --git a/.gitignore b/.gitignore index 5e3dfbb..13245df 100644 --- a/.gitignore +++ b/.gitignore @@ -10,3 +10,4 @@ dist/ build/ *egg-info *~ +.venv diff --git a/BUILD.rst b/BUILD.rst index c0e7aee..6df9453 100644 --- a/BUILD.rst +++ b/BUILD.rst @@ -1,40 +1,173 @@ Releasing StatusChecker ======================= -1. Execute tests using different Python implementations and versions. - See ``_ for instructions. -2. Set ``$VERSION`` shell variable to ease copy-pasting further commands:: +Using Invoke +~~~~~~~~~~~~ - VERSION=x.y +Invoke tasks are defined in the ``_ file and they are executed from +the command line like:: -3. Update ``__version__`` in ``_:: + inv[oke] task [options] - sed -i "s/__version__ = .*/__version__ = '$VERSION'/" robotstatuschecker.py - git diff # verify changes - git commit -m "Updated __version__ to $VERSION" robotstatuschecker.py && git push +Run ``invoke`` without arguments for help. All tasks can be listed using +``invoke --list`` and each task's usage with ``invoke --help task``. -4. Tag:: +Preparation +----------- - git tag -a $VERSION -m "Release $VERSION" && git push --tags +1. Check that you are on the master branch and have nothing left to commit, + pull, or push:: -5. Create distribution:: + git branch + git status + git pull --rebase + git push - python setup.py sdist register upload +2. Clean up:: -6. Verify that `PyPI pages `_ - look good. + invoke clean -7. Test that installation works:: +3. Execute tests using different Python implementations and versions. + See ``_ for instructions. - pip install robotstatuschecker --upgrade +4. Set version information to a shell variable to ease copy-pasting further + commands. Add ``aN``, ``bN`` or ``rcN`` postfix if creating a pre-release:: -8. ``__version__`` back to ``devel``:: + VERSION= - sed -i "s/__version__ = .*/__version__ = 'devel'/" robotstatuschecker.py - git diff # verify changes - git commit -m "__version__ back to devel" robotstatuschecker.py && git push + For example, ``VERSION=3.0.1`` or ``VERSION=3.1a2``. -9. Advertise on mailing lists, `Twitter `_, +Release notes +------------- + +1. Set GitHub user information into shell variables to ease copy-pasting the + following command:: + + GITHUB_USERNAME= + GITHUB_PASSWORD= + + Alternatively, supply the credentials when running that command. + +2. Generate a template for the release notes:: + + invoke release-notes -w -v $VERSION -u $GITHUB_USERNAME -p $GITHUB_PASSWORD + + The ``-v $VERSION`` option can be omitted if `version is already set + `__. Omit the ``-w`` option if you just want to get release + notes printed to the console, not written to a file. + + When generating release notes for a preview release like ``3.0.2rc1``, + the list of issues is only going to contain issues with that label + (e.g. ``rc1``) or with a label of an earlier preview release (e.g. + ``alpha1``, ``beta2``). + +2. Fill the missing details in the generated release notes template. + +3. Make sure that issues have correct information: + + - All issues should have type (bug, enhancement or task) and priority set. + Notice that issues with the task type are automatically excluded from + the release notes. + - Issue priorities should be consistent. + - Issue titles should be informative. Consistency is good here too, but + no need to overdo it. + + If information needs to be added or edited, its better to edit it in the + issue tracker than in the generated release notes. This allows re-generating + the list of issues later if more issues are added. + +4. Add, commit and push:: + + git add docs/releasenotes/robotstatuschecker-$VERSION.rst + git commit -m "Release notes for $VERSION" docs/releasenotes/robotstatuschecker-$VERSION.rst + git push + +5. Update later if necessary. Writing release notes is typically the biggest + task when generating releases, and getting everything done in one go is + often impossible. + + +Set version +----------- + +1. Set version information in ``_:: + + invoke set-version $VERSION + +2. Commit and push changes:: + + git commit -m "Updated version to $VERSION" robotstatuschecker.py + git push + + + +Tagging +------- + +1. Create an annotated tag and push it:: + + git tag -a v$VERSION -m "Release $VERSION" + git push --tags + +2. Add short release notes to GitHub's `releases page + `_ + with a link to the full release notes. + +Creating distributions +---------------------- + +1. Checkout the earlier created tag if necessary:: + + git checkout v$VERSION + + This isn't necessary if continuing right after tagging_. + +2. Cleanup (again). This removes temporary files as well as ``build`` and + ``dist`` directories:: + + invoke clean + +3. Create source distribution and universal (i.e. Python 2 and 3 compatible) + `wheel `_:: + + python setup.py sdist bdist_wheel --universal + ls -l dist + + Distributions can be tested locally if needed. + +4. Upload distributions to PyPI:: + + twine upload dist/* + +5. Verify that project the page at `PyPI + `_ + looks good. + +6. Test installation (add ``--pre`` with pre-releases):: + + pip install --upgrade robotstatuschecker + +Post actions +------------ + +1. Back to master if needed:: + + git checkout master + +2. Set dev version based on the previous version:: + + invoke set-version dev + git commit -m "Back to dev version" robotstatuschecker.py + git push + + For example, ``1.2.3`` is changed to ``1.2.4.dev1`` and ``2.0.1a1`` + to ``2.0.1a2.dev1``. + +3. Close the `issue tracker milestone + `_. + Create also new milestone for the next release unless one exists already. + +4. Advertise on mailing lists, `Twitter `_, `LinkedIn `_, and elsewhere as needed. diff --git a/README.rst b/README.rst index bf82173..aa5fcca 100644 --- a/README.rst +++ b/README.rst @@ -60,10 +60,19 @@ the word ``FAIL`` (in uppercase) somewhere in the test case documentation. The expected error message must then follow the ``FAIL`` marker. +For robotframework version 4 you can also change the expected status +to *SKIP* by adding the word ``SKIP`` in the test case documentation. +Like Fail, the expected skip message must follow the word ``SKIP``. +If a test documentation contains the words ``FAIL`` and ``SKIP``, ``SKIP`` +will be ignored and the expected status will be *FAIL*. + If a test is expected to *PASS* with a certain message, the word ``PASS`` must be added to its documentation explicitly and the expected message given after that. +If a message check should happen in test setup or teardown, that check +must be prefixed with ``SETUP`` or ``TEARDOWN`` word. + The expected message can also be specified as a regular expression by prefixing it with ``REGEXP:``. The specified regular expression must match the error message fully. Having spaces between the status, @@ -87,6 +96,11 @@ statuses and messages: [Documentation] FAIL Expected error message Steps + Check in test setup is done by SETUP marker + [Documentation] LOG SETUP This first log message in test setup + [Setup] Test specific setup + Steps + Exclude documentation before marker [Documentation] This text is ignored FAIL Expected error message Steps @@ -126,11 +140,14 @@ The part after the colon species the message. For example, ``1:2`` means the second message of the first keyword and ``1.2:3`` is the third message of the second child keyword of the first keyword. The message index is optional and defaults to ``1``. +The message index also supports wildcard ``*``. For example ``1:*`` +matches any message of the first keyword. Message level is specified before the actual message, and it can be any of the valid log levels in capital letters. If the level is not given it defaults to ``INFO``. Starting from 1.4 release also -``ERROR`` level is supported. +``ERROR`` level is supported. The message level also supports wildcard +``ANY`` which will match all log levels. Possible leading and trailing whitespace is ignored both in the expected and in the actual log message. diff --git a/docs/releasenotes/robotstatuschecker-1.5.0.rst b/docs/releasenotes/robotstatuschecker-1.5.0.rst new file mode 100644 index 0000000..703ba1a --- /dev/null +++ b/docs/releasenotes/robotstatuschecker-1.5.0.rst @@ -0,0 +1,48 @@ +======================== +robotstatuschecker 1.5.0 +======================== + + +.. default-role:: code + + +StatusChecker is a tool for validating that executed `Robot Framework`_ test cases +have expected statuses and log messages. It is mainly useful for Robot Framework +test library developers who want to use Robot Framework to also test their libraries. +StatusChecker 1.4 and newer are compatible both with Python 2 and Python 3. + +StatusChecker project is hosted at GitHub and downloads are at PyPI_ +.. _Robot Framework: http://robotframework.org +.. _PyPI: https://github.com/robotframework/statuschecker +.. _issue tracker: https://github.com/robotframework/SeleniumLibrary/issues?q=milestone%3Av1.5.0 + + +.. contents:: + :depth: 2 + :local: + +Most important enhancements +=========================== + +Broken with RF4 (`#11`_) +------------------------ +Fixes deprecation warning with RF 4.0 + +Full list of fixes and enhancements +=================================== + +.. list-table:: + :header-rows: 1 + + * - ID + - Type + - Priority + - Summary + * - `#11`_ + - bug + - critical + - Broken with RF4 + +Altogether 1 issue. View on the `issue tracker `__. + +.. _#11: https://github.com/robotframework/statuschecker/issues/11 diff --git a/docs/releasenotes/robotstatuschecker-1.5.1.rst b/docs/releasenotes/robotstatuschecker-1.5.1.rst new file mode 100644 index 0000000..a2fe74b --- /dev/null +++ b/docs/releasenotes/robotstatuschecker-1.5.1.rst @@ -0,0 +1,43 @@ +======================== +robotstatuschecker 1.5.1 +======================== + + +.. default-role:: code + + +StatusChecker is a tool for validating that executed `Robot Framework`_ test cases +have expected statuses and log messages. It is mainly useful for Robot Framework +test library developers who want to use Robot Framework to also test their libraries. +StatusChecker 1.4 and newer are compatible both with Python 2 and Python 3. + +This reverts previous release because it does not work. + +StatusChecker project is hosted at GitHub and downloads are at PyPI_ +.. _Robot Framework: http://robotframework.org +.. _PyPI: https://github.com/robotframework/statuschecker +.. _issue tracker: https://github.com/robotframework/SeleniumLibrary/issues?q=milestone%3Av1.5.1 + + +.. contents:: + :depth: 2 + :local: + +Full list of fixes and enhancements +=================================== + +.. list-table:: + :header-rows: 1 + + * - ID + - Type + - Priority + - Summary + * - `#17`_ + - --- + - --- + - Revert RF 4.0 support because it does not work + +Altogether 1 issue. View on the `issue tracker `__. + +.. _#17: https://github.com/robotframework/statuschecker/issues/17 diff --git a/docs/releasenotes/robotstatuschecker-2.0.0.rst b/docs/releasenotes/robotstatuschecker-2.0.0.rst new file mode 100644 index 0000000..34f2f32 --- /dev/null +++ b/docs/releasenotes/robotstatuschecker-2.0.0.rst @@ -0,0 +1,77 @@ +======================== +robotstatuschecker 2.0.0 +======================== + + +.. default-role:: code + + +StatusChecker is a tool for validating that executed `Robot Framework`_ test cases +have expected statuses and log messages. It is mainly useful for Robot Framework +test library developers who want to use Robot Framework to also test their libraries. +StatusChecker 2 and newer are compatible both with Python 3.6+ and Robot Framework 3.2 +4.0. + +StatusChecker project is hosted at GitHub and downloads are at PyPI_ +.. _Robot Framework: http://robotframework.org +.. _PyPI: https://github.com/robotframework/statuschecker +.. _issue tracker: https://github.com/robotframework/SeleniumLibrary/issues?q=milestone%3Av2.0.0 + + +.. contents:: + :depth: 2 + :local: + +Most important enhancements +=========================== + +Broken with RF4 (`#11`_) +------------------------ +Status checker is not compatible with RF 4.0 and does not anymore display +warning when it uses RF result parser. + +Add SETUP and TEARDOWN markers for checking logs from test setup and teardown (`#21`_) +-------------------------------------------------------------------------------------- +In previous releases keywords in setup and teardown could have been referenced +index starting from 1. If setup was present, then index one would point the +setup keyword. But if setup was not present, index 1 would point to the first +keyword in test body. This is somewhat confusing and now keywords is setup and +teardown must be targeted by using SETUP and TEARDOWN markers. Also index one +will always point to the first keyword in the test body. + +Backwards incompatible changes +============================== + +Drop Python 2 support and use Python 3.6 is minimum version. (`#14`_) +---------------------------------------------------------------------- +This release drops support for Python 2 and raises minimum version Python to +3.6. Also Robot Framework 3.2 and 4.0 are supported by this release. + +Full list of fixes and enhancements +=================================== + +.. list-table:: + :header-rows: 1 + + * - ID + - Type + - Priority + - Summary + * - `#11`_ + - bug + - critical + - Broken with RF4 + * - `#14`_ + - enhancement + - critical + - Drop Python 2 support and use Python 3.6 is minimum version. + * - `#21`_ + - enhancement + - high + - Add SETUP and TEARDOWN markers for checking logs from test setup and teardown + +Altogether 3 issues. View on the `issue tracker `__. + +.. _#11: https://github.com/robotframework/statuschecker/issues/11 +.. _#14: https://github.com/robotframework/statuschecker/issues/14 +.. _#21: https://github.com/robotframework/statuschecker/issues/21 diff --git a/docs/releasenotes/robotstatuschecker-2.0.1.rst b/docs/releasenotes/robotstatuschecker-2.0.1.rst new file mode 100644 index 0000000..e784cd0 --- /dev/null +++ b/docs/releasenotes/robotstatuschecker-2.0.1.rst @@ -0,0 +1,51 @@ +======================== +robotstatuschecker 2.0.1 +======================== + + +.. default-role:: code + + +StatusChecker is a tool for validating that executed `Robot Framework`_ test cases +have expected statuses and log messages. It is mainly useful for Robot Framework +test library developers who want to use Robot Framework to also test their libraries. +StatusChecker 2.0.1 is compatible with Python 3.6+ and RF 3.2.2 and RF 4.0.1. StatusChecker +2.0.1 fixes regression with Rf 3.2.2 when test has test setup. + +StatusChecker project is hosted at GitHub and downloads are at PyPI_ +.. _Robot Framework: http://robotframework.org +.. _PyPI: https://github.com/robotframework/statuschecker +.. _issue tracker: https://github.com/robotframework/SeleniumLibrary/issues?q=milestone%3Av2.0.1 + + +.. contents:: + :depth: 2 + :local: + +Most important enhancements +=========================== + +statuschecker works different with different RF versions (`#26`_) +----------------------------------------------------------------- +Release 2.0.0 added support for SETUP and TEARDOWN markers, but also +introduced regression when test had test setup and log messages +where checked. + +Full list of fixes and enhancements +=================================== + +.. list-table:: + :header-rows: 1 + + * - ID + - Type + - Priority + - Summary + * - `#26`_ + - bug + - critical + - statuschecker works different with different RF versions + +Altogether 1 issue. View on the `issue tracker `__. + +.. _#26: https://github.com/robotframework/statuschecker/issues/26 diff --git a/docs/releasenotes/robotstatuschecker-2.0.2.rst b/docs/releasenotes/robotstatuschecker-2.0.2.rst new file mode 100644 index 0000000..bf85676 --- /dev/null +++ b/docs/releasenotes/robotstatuschecker-2.0.2.rst @@ -0,0 +1,50 @@ +======================== +robotstatuschecker 2.0.2 +======================== + + +.. default-role:: code + + +StatusChecker is a tool for validating that executed `Robot Framework`_ test cases +have expected statuses and log messages. It is mainly useful for Robot Framework +test library developers who want to use Robot Framework to also test their libraries. +StatusChecker 2.0.1 is compatible with Python 3.6+ and RF 3.2.2 and RF 4.0.1. StatusChecker +2.0.1 fixes regression with Rf 3.2.2 when test has test setup. + +StatusChecker project is hosted at GitHub and downloads are at PyPI_ +.. _Robot Framework: http://robotframework.org +.. _PyPI: https://github.com/robotframework/statuschecker +.. _issue tracker: https://github.com/robotframework/SeleniumLibrary/issues?q=milestone%3Av2.0.2 + + +.. contents:: + :depth: 2 + :local: + +Most important enhancements +=========================== + +Keywords teardown in RF 3 could be accessed without TEARDOWN marker (`#28`_) +---------------------------------------------------------------------------- +Keywords teardown in RF 3 could be accessed without TEARDOWN marker. It as +used as is, but it should raise an error. + +Full list of fixes and enhancements +=================================== + +.. list-table:: + :header-rows: 1 + + * - ID + - Type + - Priority + - Summary + * - `#28`_ + - bug + - critical + - Keywords teardown in RF 3 could be accessed without TEARDOWN marker + +Altogether 1 issue. View on the `issue tracker `__. + +.. _#28: https://github.com/robotframework/statuschecker/issues/28 diff --git a/docs/releasenotes/robotstatuschecker-2.0.3.rst b/docs/releasenotes/robotstatuschecker-2.0.3.rst new file mode 100644 index 0000000..e7a8fb8 --- /dev/null +++ b/docs/releasenotes/robotstatuschecker-2.0.3.rst @@ -0,0 +1,50 @@ +======================== +robotstatuschecker 2.0.3 +======================== + + +.. default-role:: code + + +StatusChecker is a tool for validating that executed `Robot Framework`_ test cases +have expected statuses and log messages. It is mainly useful for Robot Framework +test library developers who want to use Robot Framework to also test their libraries. +StatusChecker 2.0.1 is compatible with Python 3.6+ and RF 3.2.2 and RF 4.0.1. StatusChecker +2.0.2 fixes regression with 2.0.2 release + +StatusChecker project is hosted at GitHub and downloads are at PyPI_ + +.. _Robot Framework: http://robotframework.org +.. _PyPI: https://github.com/robotframework/statuschecker +.. _issue tracker: https://github.com/robotframework/SeleniumLibrary/issues?q=milestone%3Av2.0.3 + + +.. contents:: + :depth: 2 + :local: + +Most important enhancements +=========================== + +**EXPLAIN** or remove these. + +- Revert changes in release 2.0.2, it causes too much other failures (`#29`_) + +Full list of fixes and enhancements +=================================== + +.. list-table:: + :header-rows: 1 + + * - ID + - Type + - Priority + - Summary + * - `#29`_ + - bug + - critical + - Revert changes in release 2.0.2, it causes too much other failures + +Altogether 1 issue. View on the `issue tracker `__. + +.. _#29: https://github.com/robotframework/statuschecker/issues/29 diff --git a/docs/releasenotes/robotstatuschecker-2.1.0.rst b/docs/releasenotes/robotstatuschecker-2.1.0.rst new file mode 100644 index 0000000..a9e98b1 --- /dev/null +++ b/docs/releasenotes/robotstatuschecker-2.1.0.rst @@ -0,0 +1,50 @@ +======================== +robotstatuschecker 2.1.0 +======================== + + +.. default-role:: code + + +StatusChecker is a tool for validating that executed `Robot Framework`_ test cases +have expected statuses and log messages. It is mainly useful for Robot Framework +test library developers who want to use Robot Framework to also test their libraries. +StatusChecker 1.4 and newer are compatible with Python 3.6+. + +StatusChecker project is hosted at GitHub and downloads are at PyPI_ +.. _Robot Framework: http://robotframework.org +.. _PyPI: https://github.com/robotframework/statuschecker +.. _issue tracker: https://github.com/robotframework/SeleniumLibrary/issues?q=milestone%3Av2.1.0 + + +.. contents:: + :depth: 2 + :local: + +Most important enhancements +=========================== + +Wildcard support for the log verification (`#32`_) +-------------------------------------------------- +Now it possible to verify keyword log messages with a wildcard (*), example with 1:*. +This allows to verify that log massage is somewhere in the keyword, but specific index +of the log message is not mandatory anymore. + +Full list of fixes and enhancements +=================================== + +.. list-table:: + :header-rows: 1 + + * - ID + - Type + - Priority + - Summary + * - `#32`_ + - enhancement + - high + - Wildcard support for the log verification + +Altogether 1 issue. View on the `issue tracker `__. + +.. _#32: https://github.com/robotframework/statuschecker/issues/32 diff --git a/docs/releasenotes/robotstatuschecker-2.2.0.rst b/docs/releasenotes/robotstatuschecker-2.2.0.rst new file mode 100644 index 0000000..c344d62 --- /dev/null +++ b/docs/releasenotes/robotstatuschecker-2.2.0.rst @@ -0,0 +1,42 @@ +======================== +robotstatuschecker 2.2.0 +======================== + + +.. default-role:: code + + +StatusChecker is a tool for validating that executed `Robot Framework`_ test cases +have expected statuses and log messages. It is mainly useful for Robot Framework +test library developers who want to use Robot Framework to also test their libraries. +StatusChecker is compatible both with Python 3.6+. This release add support for RF 4 +SKIP status. + +StatusChecker project is hosted at GitHub and downloads are at PyPI_ +.. _Robot Framework: http://robotframework.org +.. _PyPI: https://github.com/robotframework/statuschecker +.. _issue tracker: https://github.com/robotframework/SeleniumLibrary/issues?q=milestone%3Av2.2.0 + + +.. contents:: + :depth: 2 + :local: + +Full list of fixes and enhancements +=================================== + +.. list-table:: + :header-rows: 1 + + * - ID + - Type + - Priority + - Summary + * - `#36`_ + - --- + - --- + - Support RF 4 style SKIP + +Altogether 1 issue. View on the `issue tracker `__. + +.. _#36: https://github.com/robotframework/statuschecker/issues/36 diff --git a/pyproject.toml b/pyproject.toml new file mode 100644 index 0000000..021cb23 --- /dev/null +++ b/pyproject.toml @@ -0,0 +1,3 @@ +[tool.black] +line-length = 100 +target-version = ['py36'] diff --git a/requirements-dev.txt b/requirements-dev.txt new file mode 100644 index 0000000..ed0b8ed --- /dev/null +++ b/requirements-dev.txt @@ -0,0 +1,8 @@ +invoke >= 1.4.1 +rellu >= 0.6 +twine >= 3.4.1 +wheel >= 0.36.2 +black >= 21.9b0 +flake8 >= 3.9.2 +robotframework-tidy >= 1.6.1 +isort >= 5.9.3 \ No newline at end of file diff --git a/robotstatuschecker.py b/robotstatuschecker.py index d1f04c8..9f90a8b 100755 --- a/robotstatuschecker.py +++ b/robotstatuschecker.py @@ -27,27 +27,34 @@ Command-line usage: - python -m robotstatuschecker infile [outfile] + python -m robotstatuschecker infile [outfile] [--quiet] + robot --prerebotmodifier robotstatuschecker.StatusChecker data_sources + rebot --prerebotmodifier robotstatuschecker.StatusChecker robot_outputs Programmatic usage: from robotstatuschecker import process_output - process_output('infile.xml', 'outfile.xml') + process_output('infile.xml', 'outfile.xml', verbose=True) If an output file is not given, the input file is edited in place. -""" -from __future__ import print_function +By default status checker prints logging to the console. To suppress console +logging use --quiet on the command line, +--prerebotmodifier robotstatuschecker.StatusChecker:False in conjunction with +robot or rebot and verbose=False with the process_output function +""" -from os.path import abspath import re import sys +from os.path import abspath -from robot.api import ExecutionResult, ResultVisitor +from robot import __version__ as rf_version +from robot.api import ExecutionResult, ResultVisitor, logger +from robot.output import LOGGER from robot.utils import Matcher - -__version__ = 'devel' +__version__ = "2.2.1.dev1" +RF3 = rf_version.startswith("3") def process_output(inpath, outpath=None, verbose=True): @@ -64,126 +71,174 @@ def process_output(inpath, outpath=None, verbose=True): int: Number of failed critical tests after post-processing. """ if verbose: - print('Checking %s' % abspath(inpath)) - result = StatusChecker().process_output(inpath, outpath) + logger.console(f"Checking {abspath(inpath)}") + result = StatusChecker(verbose).process_output(inpath, outpath) if verbose and outpath: - print('Output: %s' % abspath(outpath)) + logger.console(f"Output: {abspath(outpath)}") return result.return_code class StatusChecker(ResultVisitor): + def __init__(self, verbose=True): + self.verbose = verbose + if str(self.verbose).lower() == 'false': + self.verbose = False + if self.verbose: + width = 78 + suite_separator = '%s' % ('=' * width) + logger.console(suite_separator) + logger.console('************** Post-processing of results by status checker.... **************') + logger.console(suite_separator) + def process_output(self, inpath, outpath=None): result = ExecutionResult(inpath) result.suite.visit(self) result.save(outpath) return result + def start_suite(self, suite): + if self.verbose: + LOGGER.start_suite(suite) + + def end_suite(self, suite): + if self.verbose: + LOGGER.end_suite(suite) + def visit_test(self, test): - expected = Expected(test.doc) - if TestStatusChecker(expected).check(test): - LogMessageChecker(expected).check(test) + if self.start_test(test) is not False: + expected = Expected(test.doc) + if TestStatusChecker(expected).check(test): + LogMessageChecker(expected).check(test) + self.end_test(test) + + def start_test(self, test): + if self.verbose: + LOGGER.start_test(test) + + def end_test(self, test): + if self.verbose: + LOGGER.end_test(test) def visit_keyword(self, kw): pass -class Expected(object): - +class Expected: def __init__(self, doc): self.status = self._get_status(doc) self.message = self._get_message(doc) self.logs = self._get_logs(doc) def _get_status(self, doc): - return 'FAIL' if 'FAIL' in doc else 'PASS' + if RF3: + return "FAIL" if "FAIL" in doc else "PASS" + if "FAIL" not in doc: + return "SKIP" if "SKIP" in doc else "PASS" + return "FAIL" def _get_message(self, doc): - if 'FAIL' not in doc and 'PASS' not in doc: - return '' + if RF3: + if "FAIL" not in doc and "PASS" not in doc: + return "" + if all(status not in doc for status in ["FAIL", "SKIP", "PASS"]): + return "" status = self._get_status(doc) - return doc.split(status, 1)[1].split('LOG', 1)[0].strip() + return doc.split(status, 1)[1].split("LOG", 1)[0].strip() def _get_logs(self, doc): - return [ExpectedLog(item) for item in doc.split('LOG')[1:]] + return [ExpectedLog(item) for item in doc.split("LOG")[1:]] -class ExpectedLog(object): - +class ExpectedLog: def __init__(self, doc): - index, message = doc.strip().split(' ', 1) - self.kw_index, self.msg_index = self._split_index(index) + index, message = doc.strip().split(" ", 1) + test_setup, kw_index, msg_index, test_teardown = self._split_index(index) + self.test_setup = test_setup + self.kw_index = kw_index + self.msg_index = msg_index + self.test_teardown = test_teardown self.level, self.message = self._split_level(message) + self.visited_setup = False @property def kw_index_str(self): - return '.'.join(str(index + 1) for index in self.kw_index) + return ".".join(str(index + 1) for index in self.kw_index) @property def msg_index_str(self): - return str(self.msg_index + 1) + return str(self.msg_index + 1) if isinstance(self.msg_index, int) else self.msg_index def _split_index(self, index): - if ':' in index: - kw_index, msg_index = index.split(':') + if ":" in index: + kw_index, msg_index = index.split(":") else: kw_index, msg_index = index, 1 - kw_index = [int(index) - 1 for index in kw_index.split('.')] - msg_index = int(msg_index) - 1 - return kw_index, msg_index + new_kw_index = [] + test_setup = False + test_teardown = False + for index in kw_index.split("."): + if index.upper() == "SETUP": + test_setup = True + new_kw_index.append(0) + elif index.upper() == "TEARDOWN": + test_teardown = True + new_kw_index.append(-1) + else: + new_kw_index.append(int(index) - 1) + msg_index = "*" if msg_index == "*" else int(msg_index) - 1 + return test_setup, new_kw_index, msg_index, test_teardown def _split_level(self, message): - for level in ['TRACE', 'DEBUG', 'INFO', 'WARN', 'FAIL', 'ERROR']: + for level in ["TRACE", "DEBUG", "INFO", "WARN", "FAIL", "ERROR", "ANY"]: if message.startswith(level): - return level, message[len(level):].strip() - return 'INFO', message - + return level, message[len(level) :].strip() + return "INFO", message -class BaseChecker(object): +class BaseChecker: def _message_matches(self, actual, expected): if actual == expected: return True - if expected.startswith('REGEXP:'): - pattern = '^%s$' % expected.replace('REGEXP:', '', 1).strip() + if expected.startswith("REGEXP:"): + pattern = f"^{expected.replace('REGEXP:', '', 1).strip()}$" if re.match(pattern, actual, re.DOTALL): return True - if expected.startswith('GLOB:'): - pattern = expected.replace('GLOB:', '', 1).strip() + if expected.startswith("GLOB:"): + pattern = expected.replace("GLOB:", "", 1).strip() matcher = Matcher(pattern, caseless=False, spaceless=False) if matcher.match(actual): return True - if expected.startswith('STARTS:'): - start = expected.replace('STARTS:', '', 1).strip() + if expected.startswith("STARTS:"): + start = expected.replace("STARTS:", "", 1).strip() if actual.startswith(start): return True return False - def _assert(self, condition, test, message): + def _assert(self, condition, test, message, fail=True): if not condition: - return self._fail(test, message) + return self._fail(test, message) if fail else False return True def _fail(self, test, message): - test.status = 'FAIL' + test.status = "FAIL" self._set_message(test, message) return False def _pass(self, test, message): - test.status = 'PASS' + test.status = "PASS" self._set_message(test, message) return True def _set_message(self, test, message): if test.message: - original = '\n\nOriginal message:\n%s' % test.message + original = f"\n\nOriginal message:\n{test.message}" else: - original = '' + original = "" test.message = message + original class TestStatusChecker(BaseChecker): - def __init__(self, expected): self.status = expected.status self.message = expected.message @@ -194,21 +249,26 @@ def check(self, test): def _check_status(self, test): condition = test.status == self.status - message = ('Test was expected to %s but it %sED.' - % (self.status, test.status)) + message = f"Test was expected to {self.status} but it {test.status}ED." return self._assert(condition, test, message) def _check_message(self, test): if not self._message_matches(test.message, self.message): - message = 'Wrong message.\n\nExpected:\n%s' % self.message + message = f"Wrong message.\n\nExpected:\n{self.message}" return self._fail(test, message) - if test.status == 'FAIL': - return self._pass(test, 'Test failed as expected.') + if test.status == "FAIL": + return self._pass(test, "Test failed as expected.") return True class LogMessageChecker(BaseChecker): + _no_setup_message = "Expected test {} to have setup but setup is not present." + _no_teardown_message = "Expected test {} to have teardown but teardown is not present." + _teardown_access_message = ( + "In test '{}' keyword is in teardown but " "was expected to ne in test body index {}" + ) + def __init__(self, expected): self.logs = expected.logs @@ -222,49 +282,114 @@ def _get_keyword(self, test, expected): kw = None try: for index in expected.kw_index: - kw = (kw or test).keywords[index] + kw = self._get_keyword_rf3_rf4(test, expected, kw, index) + if kw is None: + return kw return kw except IndexError: - message = "No keyword with index '%s'." % expected.kw_index_str + message = f"No keyword with index '{expected.kw_index_str}'." self._fail(test, message) return None - def _check_message(self, test, kw, expected): + def _get_keyword_rf3_rf4(self, test, expected, kw, index): + if RF3: + return self._get_keyword_rf3(test, expected, kw, index) + return self._get_keyword_rf4(test, expected, kw, index) + + def _get_keyword_rf3(self, test, expected, kw, index): + if expected.test_setup and not test.keywords.setup: + self._fail(test, self._no_setup_message.format(test.name)) + return None + if expected.test_teardown and not test.keywords.teardown: + self._fail(test, self._no_teardown_message.format(test.name)) + return None + if test.keywords.setup and not expected.test_setup and not expected.visited_setup: + index += 1 + expected.visited_setup = True + return (kw or test).keywords[index] + + def _get_keyword_rf4(self, test, expected, kw, index): + if expected.test_setup and not test.setup: + self._fail(test, self._no_setup_message.format(test.name)) + return None + if expected.test_teardown and not test.teardown: + self._fail(test, self._no_teardown_message.format(test.name)) + return None + if expected.test_setup and not kw: + kw = test.setup + elif expected.test_teardown and not kw: + kw = test.teardown + else: + kw = (kw or test).body[index] + return kw + + def _check_message_by_index(self, test, kw, expected): try: msg = kw.messages[expected.msg_index] except IndexError: - condition = expected.message == 'NONE' + condition = expected.message == "NONE" message = ( - "Keyword '%s' (index %s) does not have message %s." - % (kw.name, expected.kw_index_str, expected.msg_index_str)) + f"Keyword '{kw.name}' (index {expected.kw_index_str}) does not " + f"have message {expected.msg_index_str}." + ) self._assert(condition, test, message) else: if self._check_msg_level(test, kw, msg, expected): self._check_msg_message(test, kw, msg, expected) - def _check_msg_level(self, test, kw, msg, expected): - condition = msg.level == expected.level - message = ("Keyword '%s' (index %s) message %s has wrong level.\n\n" - "Expected: %s\nActual: %s" - % (kw.name, expected.kw_index_str, expected.msg_index_str, - expected.level, msg.level)) - return self._assert(condition, test, message) + def _check_message_by_wildcard(self, test, kw, expected): + if expected.message == "NONE": + message = "Message index wildcard '*' is not supported with expected message 'NONE'." + self._fail(test, message) + return - def _check_msg_message(self, test, kw, msg, expected): + for msg in kw.messages: + if self._check_msg_message(test, kw, msg, expected, fail=False): + if self._check_msg_level(test, kw, msg, expected, fail=False): + break + else: + message = ( + f"Keyword '{kw.name}' (index {expected.kw_index_str}) does not contain any logs " + f"with level {expected.level} and message '{expected.message}'." + ) + self._fail(test, message) + + def _check_message(self, test, kw, expected): + if expected.msg_index != "*": + self._check_message_by_index(test, kw, expected) + else: + self._check_message_by_wildcard(test, kw, expected) + + def _check_msg_level(self, test, kw, msg, expected, fail=True): + condition = msg.level == expected.level if expected.level != "ANY" else True + message = ( + f"Keyword '{kw.name}' (index {expected.kw_index_str}) " + f"message {expected.msg_index_str} has wrong level." + f"\n\nExpected: {expected.level}\nActual: {msg.level}" + ) + return self._assert(condition, test, message, fail) + + def _check_msg_message(self, test, kw, msg, expected, fail=True): condition = self._message_matches(msg.message.strip(), expected.message) - message = ("Keyword '%s' (index %s) message %s has wrong content.\n\n" - "Expected:\n%s\n\nActual:\n%s" - % (kw.name, expected.kw_index_str, expected.msg_index_str, - expected.message, msg.message)) - return self._assert(condition, test, message) + message = ( + f"Keyword '{kw.name}' (index {expected.kw_index_str}) " + f"message {expected.msg_index_str} has wrong content." + f"\n\nExpected:\n{expected.message}\n\nActual:\n{msg.message}" + ) + return self._assert(condition, test, message, fail) -if __name__ == '__main__': - if '-h' in sys.argv or '--help' in sys.argv: +if __name__ == "__main__": + if "-h" in sys.argv or "--help" in sys.argv: print(__doc__) sys.exit(251) + args = sys.argv[1:] try: - rc = process_output(*sys.argv[1:]) + if '--quiet' in args: + args.remove('--quiet') + rc = process_output(*args, verbose=False) + else: + rc = process_output(*args) except TypeError: print(__doc__) sys.exit(252) diff --git a/setup.py b/setup.py index 59a1eb3..7034d08 100755 --- a/setup.py +++ b/setup.py @@ -1,9 +1,6 @@ #!/usr/bin/env python -try: - from setuptools import setup -except ImportError: - from distutils.core import setup +from setuptools import setup from os.path import abspath, dirname, join import re @@ -13,14 +10,13 @@ Development Status :: 5 - Production/Stable License :: OSI Approved :: Apache Software License Operating System :: OS Independent -Programming Language :: Python :: 2 Programming Language :: Python :: 3 Topic :: Software Development :: Testing Framework :: Robot Framework """.strip().splitlines() CURDIR = dirname(abspath(__file__)) with open(join(CURDIR, NAME+'.py')) as f: - VERSION = re.search("\n__version__ = '(.*)'\n", f.read()).group(1) + VERSION = re.search('\n__version__ = "(.*)"\n', f.read()).group(1) with open(join(CURDIR, 'README.rst')) as f: README = f.read() @@ -39,5 +35,6 @@ platforms = 'any', classifiers = CLASSIFIERS, py_modules = ['robotstatuschecker'], - install_requires = ['robotframework'] + install_requires = ['robotframework'], + python_requires = '>=3.6,<4.0' ) diff --git a/tasks.py b/tasks.py new file mode 100644 index 0000000..2940d24 --- /dev/null +++ b/tasks.py @@ -0,0 +1,109 @@ +import sys +from pathlib import Path + +from invoke import task +from rellu import ReleaseNotesGenerator, Version, initialize_labels +from rellu.tasks import clean # noqa + +VERSION_PATTERN = '__version__ = "(.*)"' +REPOSITORY = "robotframework/statuschecker" +VERSION_PATH = Path("robotstatuschecker.py") +RELEASE_NOTES_PATH = Path("docs/releasenotes/robotstatuschecker-{version}.rst") +RELEASE_NOTES_TITLE = "robotstatuschecker {version}" +RELEASE_NOTES_INTRO = """ +StatusChecker is a tool for validating that executed `Robot Framework`_ test cases +have expected statuses and log messages. It is mainly useful for Robot Framework +test library developers who want to use Robot Framework to also test their libraries. +StatusChecker 1.4 and newer are compatible both with Python 2 and Python 3. + +StatusChecker project is hosted at GitHub and downloads are at PyPI_ +.. _Robot Framework: http://robotframework.org +.. _PyPI: https://github.com/robotframework/statuschecker +.. _issue tracker: https://github.com/robotframework/SeleniumLibrary/issues?q=milestone%3A{version.milestone} +""" # noqa + + +@task +def set_version(ctx, version): + """Set project version in `robotstatuschecker.py`` file. + Args: + version: Project version to set or ``dev`` to set development version. + Following PEP-440 compatible version numbers are supported: + - Final version like 3.0 or 3.1.2. + - Alpha, beta or release candidate with ``a``, ``b`` or ``rc`` postfix, + respectively, and an incremented number like 3.0a1 or 3.0.1rc1. + - Development version with ``.dev`` postfix and an incremented number like + 3.0.dev1 or 3.1a1.dev2. + When the given version is ``dev``, the existing version number is updated + to the next suitable development version. For example, 3.0 -> 3.0.1.dev1, + 3.1.1 -> 3.1.2.dev1, 3.2a1 -> 3.2a2.dev1, 3.2.dev1 -> 3.2.dev2. + """ + version = Version(version, VERSION_PATH, VERSION_PATTERN) + version.write() + print(version) + + +@task +def print_version(ctx): + """Print the current project version.""" + print(Version(path=VERSION_PATH, pattern=VERSION_PATTERN)) + + +@task +def release_notes(ctx, version=None, username=None, password=None, write=False): + """Generates release notes based on issues in the issue tracker. + Args: + version: Generate release notes for this version. If not given, + generated them for the current version. + username: GitHub username. + password: GitHub password. + write: When set to True, write release notes to a file overwriting + possible existing file. Otherwise just print them to the + terminal. + Username and password can also be specified using ``GITHUB_USERNAME`` and + ``GITHUB_PASSWORD`` environment variable, respectively. If they aren't + specified at all, communication with GitHub is anonymous and typically + pretty slow. + """ + version = Version(version, VERSION_PATH, VERSION_PATTERN) + folder = RELEASE_NOTES_PATH.parent.resolve() + folder.mkdir(parents=True, exist_ok=True) + file = RELEASE_NOTES_PATH if write else sys.stdout + generator = ReleaseNotesGenerator(REPOSITORY, RELEASE_NOTES_TITLE, RELEASE_NOTES_INTRO) + generator.generate(version, username, password, file) + + +@task +def init_labels(ctx, username=None, password=None): + """Initialize project by setting labels in the issue tracker. + Args: + username: GitHub username. + password: GitHub password. + Username and password can also be specified using ``GITHUB_USERNAME`` and + ``GITHUB_PASSWORD`` environment variable, respectively. + Should only be executed once when taking ``rellu`` tooling to use or + when labels it uses have changed. + """ + initialize_labels(REPOSITORY, username, password) + + +@task +def lint(ctx): + """Run linters + + Flake8, Black and robotframework-tidy + """ + ctx.run("black --config pyproject.toml tasks.py robotstatuschecker.py") + ctx.run("flake8 --config .flake8 tasks.py robotstatuschecker.py") + ctx.run("isort tasks.py robotstatuschecker.py") + tidy_command = [ + "robotidy", + "--lineseparator", + "unix", + "--configure", + "NormalizeAssignments:equal_sign_type=space_and_equal_sign", + "--configure", + "NormalizeAssignments:equal_sign_type_variables=space_and_equal_sign", + "test", + ] + ctx.run(" ".join(tidy_command)) diff --git a/test/README.rst b/test/README.rst index 510f917..0aed144 100644 --- a/test/README.rst +++ b/test/README.rst @@ -17,3 +17,12 @@ other tests should pass. Test statuses and messages set by StatusChecker are verified by ``run.py``. Expected statuses and messages are logged by ``Status`` keyword that all tests must use as their first keyword. + +Test cases that use features only available in RF4+ (e.g. SKIP status) should +be tagged with + +.. sourcecode:: robotframework + + [Tags] rf3unsupported + +and will be excluded in run.py if the installed robotframework is RF3. \ No newline at end of file diff --git a/test/run.py b/test/run.py index 1cb3341..0ca2664 100755 --- a/test/run.py +++ b/test/run.py @@ -1,11 +1,9 @@ #!/usr/bin/env python -from __future__ import print_function - +import sys from os.path import abspath, dirname, exists, join from platform import python_implementation, python_version from shutil import rmtree -import sys from robot import run, rebot from robot.api import ExecutionResult, ResultVisitor @@ -13,7 +11,10 @@ CURDIR = dirname(abspath(__file__)) sys.path.insert(0, dirname(CURDIR)) -from robotstatuschecker import process_output +from robot.version import VERSION # noqa + +from robotstatuschecker import process_output # noqa +from robotstatuschecker import RF3 # noqa def check_tests(robot_file): @@ -22,23 +23,25 @@ def check_tests(robot_file): checker = StatusCheckerChecker() result.suite.visit(checker) checker.print_status() + print(f"Robot Framework version: {VERSION}") sys.exit(len(checker.errors)) def _run_tests_and_process_output(robot_file): - results = join(CURDIR, 'results') - output = join(results, 'output.xml') + results = join(CURDIR, "results") + output = join(results, "output.xml") if exists(results): rmtree(results) - run(join(CURDIR, robot_file), output=output, log=None, report=None, - loglevel='DEBUG') + if RF3: + run(join(CURDIR, robot_file), output=output, log=None, report=None, loglevel="DEBUG", exclude="rf3unsupported") + else: + run(join(CURDIR, robot_file), output=output, log=None, report=None, loglevel="DEBUG") process_output(output) rebot(output, outputdir=results) return output class StatusCheckerChecker(ResultVisitor): - def __init__(self): self.errors = [] self.tests = 0 @@ -46,33 +49,48 @@ def __init__(self): def visit_test(self, test): self.tests += 1 status, message = self._get_expected(test) - errors = [self._verify(test.status, status, 'status'), - self._verify(test.message, message, 'message')] - errors = ['- %s' % e for e in errors if e] + errors = [ + self._verify(test.status, status, "status"), + self._verify(test.message, message, "message"), + ] + errors = ["- %s" % e for e in errors if e] if errors: - self.errors.append('%s:\n%s' % (test.name, '\n'.join(errors))) + self.errors.append("%s:\n%s" % (test.name, "\n".join(errors))) def _get_expected(self, test): + if RF3: + return self._get_expected_rf3(test) + return self._get_expected_rf4(test) + + def _get_expected_rf4(self, test): + if len(test.setup.body) == 0: + kw = test.body[0] + else: + kw = test.setup + return (kw.body[1].messages[0].message, kw.body[2].messages[0].message) + + def _get_expected_rf3(self, test): kw = test.keywords[0] - assert kw.name == 'Status', "No 'Status' keyword found." - return (kw.keywords[1].messages[0].message, - kw.keywords[2].messages[0].message) + assert kw.name == "Status", "No 'Status' keyword found." + return (kw.keywords[1].messages[0].message, kw.keywords[2].messages[0].message) def _verify(self, actual, expected, explanation): if actual == expected: - return '' - return ('Expected %s to be "%s" but it was "%s".' - % (explanation, expected, actual)) + return "" + return 'Expected %s to be "%s" but it was "%s".' % (explanation, expected, actual) def print_status(self): print() if self.errors: - print('%d/%d test failed:' % (len(self.errors), self.tests)) - print('\n-------------------------------------\n'.join(self.errors)) + print("%d/%d test failed:" % (len(self.errors), self.tests)) + print("\n-------------------------------------\n".join(self.errors)) else: - print('All %d tests passed/failed/logged as expected.' % self.tests) - print('Run on %s %s.' % (python_implementation(), python_version())) + if RF3: + print("All %d tests passed/failed/logged as expected." % self.tests) + else: + print("All %d tests passed/failed/logged/skipped as expected." % self.tests) + print("Run on %s %s." % (python_implementation(), python_version())) -if __name__ == '__main__': - check_tests('tests.robot') +if __name__ == "__main__": + check_tests("tests.robot") diff --git a/test/tests.robot b/test/tests.robot index c9672df..95926ae 100644 --- a/test/tests.robot +++ b/test/tests.robot @@ -1,7 +1,6 @@ *** Settings *** -Suite Setup Log Suite setup -Suite Teardown Log Suite teardown - +Suite Setup Log Suite setup +Suite Teardown Log Suite teardown *** Test Cases *** Implicit PASS @@ -13,12 +12,32 @@ Explicit PASS with message Status PASS The message Pass Execution The message +Explicit SKIP with message + [Documentation] SKIP The message + [Tags] rf3unsupported + Status SKIP The message + Skip The message + Expected FAIL [Documentation] FAIL Expected failure Status PASS Test failed as expected.\n\n ... Original message:\nExpected failure Fail Expected failure +SKIP Plus FAIL Expected FAIL + [Documentation] SKIP FAIL Expected failure + [Tags] rf3unsupported + Status PASS Test failed as expected.\n\n + ... Original message:\nExpected failure + Fail Expected failure + +FAIL Plus SKIP Expected FAIL + [Documentation] FAIL Expected failure SKIP + [Tags] rf3unsupported + Status PASS Test failed as expected.\n\n + ... Original message:\nExpected failure SKIP + Fail Expected failure SKIP + Ignore documentation before marker [Documentation] This text is ignored. FAIL Expected failure Status PASS Test failed as expected.\n\n @@ -72,13 +91,68 @@ Trailing and leading whitespace is ignored in log messages Log ${SPACE*10}xxx${SPACE*10} Log messages deeper - [Documentation] LOG 2:1 Hello LOG 2:2 World + [Documentation] + ... LOG 2:1 Hello + ... LOG 2:2 World + ... LOG 3.1 DEBUG User Keyword + ... LOG 4.1:1 User + ... LOG 4.1:2 Keyword + ... LOG 5.1:2 DEBUG STARTS: Traceback (most recent call last): + Status PASS + Log Many Hello World + Logging User Keyword + Logging User Keyword 2 + Run Keyword And Ignore Error + ... Fail My Error Here + +Log messages deeper with setup + [Documentation] + ... LOG 1:1 Hello + ... LOG 1:2 World + ... LOG 2.1 DEBUG User Keyword + ... LOG 3.1:1 User + ... LOG 3.1:2 Keyword + ... LOG 4.1:2 DEBUG STARTS: Traceback (most recent call last): + [Setup] Status PASS + Log Many Hello World + Logging User Keyword + Logging User Keyword 2 + Run Keyword And Ignore Error + ... Fail My Error Here + +Log messages deeper with wildcard + [Documentation] + ... LOG 2:1 Hello + ... LOG 2:2 ANY World ... LOG 3.1 DEBUG User Keyword - ... LOG 4.1:1 User LOG 4.1:2 Keyword + ... LOG 4.1:* User + ... LOG 4.1:* ANY Keyword + ... LOG 5.1:* DEBUG STARTS: Traceback (most recent call last): + ... LOG 6.1:* ANY REGEXP: .*recent.* + ... LOG 6.1:* DEBUG REGEXP: .*recent.* Status PASS Log Many Hello World Logging User Keyword Logging User Keyword 2 + Run Keyword And Ignore Error + ... Fail My Error Here + Run Keyword And Ignore Error + ... Fail 'recent call' in two different log levels + +Log messages deeper with wildcard and setup + [Documentation] + ... LOG 1:1 Hello + ... LOG 1:2 ANY World + ... LOG 2.1 DEBUG User Keyword + ... LOG 3.1:* User + ... LOG 3.1:* ANY Keyword + ... LOG 4.1:* DEBUG STARTS: Traceback (most recent call last): + [Setup] Status PASS + Log Many Hello World + Logging User Keyword + Logging User Keyword 2 + Run Keyword And Ignore Error + ... Fail My Error Here Log message with REGEXP [Documentation] LOG 2 REGEXP: H[ei]l{2}o w\\w+! LOG 2 REGEXP: Hell.* @@ -106,6 +180,68 @@ NONE log message No Operation Log Message +Test Setup Check Is Done By SETUP Marker + [Documentation] ... + ... LOG SETUP:1 NONE + ... LOG SETUP.2:1 PASS + ... LOG SETUP.2 PASS + ... LOG 1:1 KALA + [Setup] Status PASS + Log KALA + +Error When No Setup + [Documentation] ... + ... LOG SETUP.1:1 PASS + ... LOG 2:1 KALA + Status FAIL Expected test Error When No Setup to have setup but setup is not present. + Log KALA + +Test Setup Check Is Done By SETUP Marker and wildcard is used + [Documentation] ... + ... LOG SETUP:1 NONE + ... LOG SETUP.2:* PASS + ... LOG SETUP.2 PASS + ... LOG 1:* HAUKI + [Setup] Status PASS + Log HAUKI + +Error When No Setup and wildcard is used + [Documentation] ... + ... LOG SETUP.1:* PASS + ... LOG 2:* KALA + Status FAIL Expected test Error When No Setup and wildcard is used to have setup but setup is not present. + Log KALA + +Test Teardown Check Is Done By TEARDOWN Marker + [Documentation] ... + ... LOG TEARDOWN:1 foobar + ... LOG TEARDOWN foobar + Status PASS + [Teardown] Log foobar + +Error When No Teardown + [Documentation] LOG TEARDOWN:1 foobar + Status FAIL Expected test Error When No Teardown to have teardown but teardown is not present. + Log KALA + +Test Teardown Check Is Done By TEARDOWN Marker and wildcard is used + [Documentation] ... + ... LOG TEARDOWN:* foobar + ... LOG TEARDOWN foobar + Status PASS + [Teardown] Log foobar + +Error When No Teardown and wildcard is used + [Documentation] LOG TEARDOWN:* foobar + Status FAIL + ... Expected test Error When No Teardown and wildcard is used to have teardown but teardown is not present. + Log KALA + +Error When NONE is used with wildcard + [Documentation] LOG 2.1:* INFO NONE + Status FAIL Message index wildcard '*' is not supported with expected message 'NONE'. + Logging User Keyword 2 + Expected FAIL and log messages [Documentation] This text is ignored. FAIL Told ya!! ... LOG 2 Failing soon! @@ -129,6 +265,13 @@ Expected PASS and log messages Log Any time now... Pass Execution Told ya!! +Expected PASS and teadown does not affect + [Documentation] This text is ignored. + ... LOG 2 Passing soon! + Status PASS ${EMPTY} + Log Passing soon! + [Teardown] Log This is logged + FAILURE: Unexpected PASS [Documentation] FAIL Expected failure does not occur Status FAIL Test was expected to FAIL but it PASSED. @@ -183,6 +326,10 @@ FAILURE: Non-existing log message Status FAIL Keyword 'BuiltIn.Log' (index 2) does not have message 2. Log Message +FAILURE: Non-existing log message wildcard + [Documentation] LOG 1:* Bogus message + Status FAIL Keyword 'Status' (index 1) does not contain any logs with level INFO and message 'Bogus message'. + *** Keywords *** Logging User Keyword Log User Keyword DEBUG