[all] Quality Improvement - Chunk 5 (#5541)

[SamuelHassine]

fix(external-import): Comprehensive fixes for M-O connectors

Summary

Comprehensive review and fix of all external-import connectors from M to O, addressing deprecated API usage, exit code issues, error handling improvements, naming conventions, and code quality.

Connectors in scope:

• M: malcore, malpedia, maltiverse, malwarebazaar, malwarebazaar-recent-additions, mandiant, microsoft-defender-incidents, microsoft-sentinel-incidents, misp, misp-feed, mitre, mitre-atlas, montysecurity-c2-tracker, mwdb
• N: nti
• O: opencsam, opencti, orange-cyberdefense

📊 Statistics

Metric	Count
Connectors Reviewed	18
Connectors Modified	12
Files Changed	20
Deprecated API Fixes	25+
Exit Code Fixes	2
Code Consistency Fixes	3
Spelling/Grammar Fixes	4
Naming Convention Fixes	10
Bug Fixes (config/logic)	2
OpenCTI Modeling Fixes	17
File Handle Fixes	3
Timezone Consistency Fixes	4

🚨 Critical Fixes

Bug Fixes

Connector	Issue	Fix
malcore	Deprecated datetime.utcfromtimestamp() in run()	Changed to datetime.fromtimestamp(timestamp, tz=timezone.utc)
malcore	Variable hash shadows built-in	Renamed to hash_value
malcore	Used exit(1) instead of sys.exit(1) in main.py	Changed to sys.exit(1) for consistency
malcore	Missing import sys in main.py	Added import sys
malpedia	Deprecated datetime.utcnow()	Changed to datetime.now(timezone.utc)
malpedia	Deprecated datetime.utcfromtimestamp() in utils	Changed to datetime.fromtimestamp(timestamp, tz=timezone.utc)
malpedia	Incorrect exit code sys.exit(0) after Exception	Changed to sys.exit(1)
malwarebazaar-recent-additions	print(e) instead of proper logging in main	Changed to traceback.print_exc()
malwarebazaar-recent-additions	Incorrect exit code sys.exit(0) in main exception	Changed to sys.exit(1)
malwarebazaar-recent-additions	Missing import traceback	Added import traceback
misp-feed	Deprecated datetime.utcfromtimestamp() (18 occurrences)	Changed to datetime.fromtimestamp(timestamp, tz=timezone.utc)
misp-feed	Missing timezone import	Added timezone to datetime import
mandiant	Typo "Failed du to connector state error"	Changed to "Failed due to connector state error"
malwarebazaar	datetime.now() not timezone-aware	Changed to datetime.now(timezone.utc)
mwdb	Comment typo "retrieve le last month"	Changed to "retrieve the last month"
mwdb	datetime.fromtimestamp() not timezone-aware	Changed to datetime.fromtimestamp(..., tz=timezone.utc)
mwdb	Misleading error message "KeyError encountered"	Changed to "Unexpected error"
mwdb	Used exit(1) instead of sys.exit(1)	Changed to sys.exit(1) for consistency
nti	Used exit(1) instead of sys.exit(1) in main.py	Changed to sys.exit(1) for consistency
nti	Missing import sys in main.py	Added import sys
nti	datetime.now() not timezone-aware	Changed to datetime.now(timezone.utc)
nti	Bare except: clause in config_loader.py	Changed to except Exception:
nti	Function create_AutonomousSystem uses PascalCase	Changed to create_autonomous_system (snake_case)
nti	Chinese comment in mapping.py	Translated to English
opencsam	Variable object shadows built-in	Renamed to obj
opencsam	datetime.now().astimezone() inconsistent pattern	Changed to datetime.now(tz=pytz.UTC)
opencti	Bug: empty config path ["config", ""] for companies_file_url	Fixed to ["config", "companies_file_url"]
opencti	Bug: missing config parameter in get_config_variable	Added config parameter
orange-cyberdefense	Variable zip shadows built-in	Renamed to zip_file
orange-cyberdefense	Variable object shadows built-in	Renamed to stix_object
orange-cyberdefense	PEP 8: if not "x" in y style	Changed to if "x" not in y (2 occurrences)

🎯 OpenCTI Modeling Fixes

STIX/OpenCTI Compliance Issues

Connector	Issue	Fix
malcore	Bug: self.identity assigned as tuple due to trailing comma	Removed trailing comma to correctly assign identity object
malcore	Indicator.generate_id(hash_value) incorrect - ID should be from pattern	Changed to Indicator.generate_id(pattern)
malcore	is_family=True for individual malware samples	Changed to is_family=False - samples are not malware families
malwarebazaar-recent-additions	Bare except: clause	Changed to except Exception:
malwarebazaar-recent-additions	Relationship related-to from Artifact→Indicator	Changed to based-on from Indicator→Artifact per STIX spec
maltiverse	Missing work_id in send_stix2_bundle() call	Added work_id=work_id parameter
mwdb	Hash pattern [file:hashes.sha256 = ...] incorrect format	Changed to [file:hashes.'SHA-256' = ...] per STIX spec
mwdb	Relationship based-on from Indicator→Malware	Changed to indicates per OpenCTI modeling
mwdb	Duplicate print() statements for errors	Removed duplicate prints, using only self.helper.log_error()
nti	URL observable object_marking_refs not a list	Changed self.tlp_marking.id to [self.tlp_marking.id] for consistency
nti	obtain_location_name() never sets location_type	Added proper location type assignment (Position, City, Administrative-Area, Country)
nti	Indicator created_by_ref=self.author passes full object	Changed to self.author["id"] for correct reference
opencsam	Config file opened without context manager	Changed to with open(...) as f: for proper resource cleanup
opencti	Config file opened without context manager	Changed to with open(...) as f: for proper resource cleanup
orange-cyberdefense	Config file opened without context manager	Changed to with open(...) as f: for proper resource cleanup
orange-cyberdefense	.astimezone() uses local timezone instead of UTC	Changed to .astimezone(datetime.timezone.utc) for consistency
orange-cyberdefense	Redundant .astimezone() after UTC datetime creation	Removed redundant conversion, using .isoformat() directly

Explanation

Indicator ID Generation

# Before - INCORRECT: ID based on hash value
id=Indicator.generate_id(hash_value)

# After - CORRECT: ID based on pattern
id=Indicator.generate_id(pattern)

Malware is_family Flag

# Before - INCORRECT: Individual samples marked as families
is_family=True

# After - CORRECT: Individual samples are not families
is_family=False

Relationship Direction (Indicator → Observable)

# Before - INCORRECT: Artifact → Indicator with related-to
fromType="Artifact", toType="Indicator", relationship_type="related-to"

# After - CORRECT: Indicator → Artifact with based-on
fromType="Indicator", toType="Artifact", relationship_type="based-on"

STIX Hash Pattern Format

# Before - INCORRECT: lowercase hash algorithm
pattern = "[file:hashes.sha256 = '" + hash + "']"

# After - CORRECT: quoted hash algorithm name
pattern = "[file:hashes.'SHA-256' = '" + hash + "']"

🐛 Bug Fixes

Deprecated DateTime APIs

Python 3.12 deprecated datetime.utcnow() and datetime.utcfromtimestamp(). These must be replaced with timezone-aware alternatives.

Affected connectors: malcore, malpedia, misp-feed

# Before - DEPRECATED in Python 3.12+
now = datetime.utcfromtimestamp(timestamp)
now = datetime.utcnow()

# After - CORRECT: timezone-aware
from datetime import datetime, timezone
now = datetime.fromtimestamp(timestamp, tz=timezone.utc)
now = datetime.now(timezone.utc)

Exit Code Corrections

Using exit(0) or sys.exit(0) in exception handlers incorrectly signals successful termination. These must return non-zero exit codes.

Affected connectors: malpedia, malwarebazaar-recent-additions

# Before - INCORRECT: signals success on error
except Exception as e:
    print(e)
    time.sleep(10)
    sys.exit(0)

# After - CORRECT: signals error with proper logging
except Exception:
    traceback.print_exc()
    sys.exit(1)

Code Consistency Fixes

Using bare exit() instead of sys.exit() is inconsistent and may not work correctly in all environments.

Affected connectors: malcore, nti, mwdb

# Before - INCONSISTENT: bare exit
exit(1)

# After - CONSISTENT: sys.exit
sys.exit(1)

🔧 Code Quality

Naming Convention Fixes

Connector	File	Original	Fixed	Reason
malcore	core.py	hash	hash_value	Shadows built-in function
malwarebazaar-recent-additions	malwarebazaar-recent-additions.py	relation_ship	create_relationship	PEP 8: inconsistent underscore placement
mandiant	utils.py	cleanhtml	clean_html	PEP 8: functions should be snake_case
nti	converter_to_stix.py	create_AutonomousSystem	create_autonomous_system	PEP 8: functions should be snake_case
opencsam	opencsam.py	object	obj	Shadows built-in function
orange-cyberdefense	orange-cyberdefense.py	zip	zip_file	Shadows built-in function
orange-cyberdefense	orange-cyberdefense.py	object	stix_object	Shadows built-in function
orange-cyberdefense	orange-cyberdefense.py	_get_ranged_scored	_get_ranged_score	Grammatically incorrect (past participle instead of noun)
orange-cyberdefense	orange-cyberdefense.py	object (param)	stix_obj	Shadows built-in in _process_object param
orange-cyberdefense	orange-cyberdefense.py	object (loop var)	stix_obj	Shadows built-in in _get_report_iocs loop

Spelling & Grammar Fixes

Connector	File	Original	Fixed
mandiant	base.py	"Failed du to connector state error"	"Failed due to connector state error"
mwdb	mwdb.py	"retrieve le last month"	"retrieve the last month"
mwdb	mwdb.py	"KeyError encountered" (wrong error type)	"Unexpected error"
nti	mapping.py	# 补充修改的反诈 (Chinese comment)	# Supplementary fraud types

📦 Connectors Modified

M (14 connectors)

Connector	Status	Changes
malcore	✅ Modified	DateTime fix, variable rename hash → hash_value, exit(1) → sys.exit(1), MODELING: tuple bug fix, Indicator ID generation fix, is_family=False
malpedia	✅ Modified	DateTime fixes (utils.py, connector.py), exit code fix
maltiverse	✅ Modified	MODELING: Added missing work_id parameter to send_stix2_bundle()
malwarebazaar	✅ Modified	datetime.now() → datetime.now(timezone.utc)
malwarebazaar-recent-additions	✅ Modified	Exit code fix, traceback.print_exc(), function rename, MODELING: bare except: fix, relationship direction/type fix (Indicator→Artifact, based-on)
mandiant	✅ Modified	Typo fix "du" → "due", function rename cleanhtml → clean_html
microsoft-defender-incidents	✅ No changes	Already using modern practices
microsoft-sentinel-incidents	✅ No changes	Already using modern practices
misp	✅ No changes	Already using modern practices
misp-feed	✅ Modified	DateTime fixes (18 occurrences)
mitre	✅ No changes	Already reviewed
mitre-atlas	✅ No changes	Already reviewed
montysecurity-c2-tracker	⚠️ Not modified	Uses direct API, needs significant refactor
mwdb	✅ Modified	DateTime fix, comment typo fix, error message fix, exit(1) → sys.exit(1), MODELING: hash pattern format fix, relationship type fix (indicates), removed print statements

N (1 connector)

Connector	Status	Changes
nti	✅ Modified	exit(1) → sys.exit(1), datetime.now() → datetime.now(timezone.utc), bare except: → except Exception:, function rename create_AutonomousSystem → create_autonomous_system, Chinese comment translated, MODELING: URL marking refs fix, location_type fix, created_by_ref fix

Naming Convention Review (Pass 2): All functions, variables, classes, files, and constants in the NTI connector follow PEP 8 naming conventions. No additional naming issues found.

OpenCTI Modeling Review (Pass 3): Fixed 3 STIX/OpenCTI compliance issues in converter_to_stix.py.

O (3 connectors)

Connector	Status	Changes
opencsam	✅ Modified	Variable object → obj, datetime.now().astimezone() → datetime.now(tz=pytz.UTC), MODELING: config file context manager fix
opencti	✅ Modified	Bug fix: empty config path and missing config parameter in get_config_variable for companies_file_url, MODELING: config file context manager fix
orange-cyberdefense	✅ Modified	Variable zip → zip_file, variable object → stix_object, PEP 8 not in style (2 occurrences), function _get_ranged_scored → _get_ranged_score, parameter object → stix_obj in _process_object, loop variable object → stix_obj in _get_report_iocs, MODELING: config file context manager fix, timezone consistency fixes (4 occurrences)

Naming Convention Review (Pass 2): All functions, variables, classes, files, and constants in opencsam and opencti connectors follow PEP 8 naming conventions. No additional naming issues found.

OpenCTI Modeling Review (Pass 3): Fixed file handle leaks (3 connectors) and timezone inconsistencies (4 occurrences in orange-cyberdefense).

📝 Files Changed Breakdown

Connector	Files Modified
malcore	src/malcore/core.py, src/main.py
malpedia	src/malpedia_connector/connector.py, src/malpedia_services/utils.py
maltiverse	src/main.py
malwarebazaar	src/malwarebazaar_connector/connector.py
malwarebazaar-recent-additions	src/malwarebazaar-recent-additions.py
mandiant	src/connector/base.py, src/connector/utils.py, src/connector/reports.py
misp-feed	src/connector/connector.py
mwdb	src/mwdb.py
nti	src/main.py, src/nti/connector.py, src/nti/config_loader.py, src/nti/converter_to_stix.py, src/nti/mapping.py
opencsam	src/opencsam.py
opencti	src/connector.py
orange-cyberdefense	src/orange-cyberdefense.py

⚠️ Notes

Connectors Not Modified

1. montysecurity-c2-tracker: This connector uses direct OpenCTI API calls instead of the standard connector helper pattern. It requires a more significant refactor to align with OpenCTI connector best practices. The connector:

   • Uses print() statements for all logging
   • Directly instantiates OpenCTIApiClient instead of using OpenCTIConnectorHelper
   • Has a recursive loop() function for error recovery
   • Should be refactored to use the standard connector pattern in a future PR

2. microsoft-defender-incidents: Already uses proper timezone-aware datetime in main connector code.

3. microsoft-sentinel-incidents: Already uses proper timezone-aware datetime.

4. misp: Already uses modern datetime practices with datetime.now(tz=timezone.utc).

5. mitre, mitre-atlas: Already reviewed and use proper patterns.

Breaking Changes

None. All changes are backwards compatible.

File Naming

No file renames were performed. The following files use hyphenated names which is acceptable for main entry point scripts:

• malwarebazaar-recent-additions/src/malwarebazaar-recent-additions.py
• orange-cyberdefense/src/orange-cyberdefense.py

✅ Testing Checklist

• All 12 modified connectors initialize without errors
• DateTime handling verified with timezone-aware timestamps
• Error handling produces correct exit codes
• black . and isort --profile=black . pass on modified files