Jekyll2025-12-24T02:59:17+00:00https://george-gca.github.io/feed.xmlblankA comprehensive guide to Python best practices2025-11-24T23:44:15+00:002025-11-24T23:44:15+00:00https://george-gca.github.io/blog/2025/python-best-practicesA guide of best practices for developing in Python. Inspired by Rui Maranhao’s gist.
defprocess_data(data,encoding='utf-8',timeout=30):"""Process data with explicit parameters."""# Parameters are clear and visible
returndata.decode(encoding)
No
defprocess_data(data):"""Process data with hidden defaults."""# Magic values hidden inside function
returndata.decode('utf-8')# What encoding? Why?
Don’t create artificial barriers or “ownership” of code. If you see a bug or improvement opportunity in any part of the codebase, fix it.
This principle comes from Khan Academy’s development philosophy.
Yes
See a typo in someone else’s module? Fix it.
Found a bug in a different team’s code? Submit a fix.
Notice outdated documentation? Update it.
No
“That’s not my module, I won’t touch it.”
“I’ll just work around this bug in their code.”
Leaving broken code for the “owner” to fix.
Fix each issue (bad design, wrong decision, or poor code) as soon as it is discovered.
Yes
# You notice during code review that a function is doing too much.
# Refactor it immediately:
defprocess_user_data(user):validate_user(user)save_to_database(user)send_welcome_email(user)
No
# You notice the problem but add a TODO comment instead:
defprocess_user_data(user):# TODO: This function does too much, should refactor
validate_user(user)save_to_database(user)send_welcome_email(user)# TODO will likely never be addressed
Don’t wait for the “perfect” solution. Ship working code, then iterate.
Yes
Implement a basic working feature, deploy it, gather feedback, improve it.
Write simple tests now rather than waiting to design the perfect test suite.
No
Endlessly debating the ideal architecture without writing code.
Waiting months to ship because you want every edge case handled.
Test ruthlessly. Write docs for new features.
Yes
defdivide(a,b):"""Divide a by b.
:param a: Numerator
:param b: Denominator (must be non-zero)
:returns: Result of a / b
:raises ValueError: If b is zero
"""ifb==0:raiseValueError("Cannot divide by zero")returna/b# And corresponding tests:
classTestDivide(unittest.TestCase):deftest_divide_positive_numbers(self):self.assertEqual(divide(10,2),5)deftest_divide_by_zero_raises_error(self):withself.assertRaises(ValueError):divide(10,0)
No
defdivide(a,b):returna/b# No docs, no tests, will crash on zero
Even more important than Test-Driven Development–Human-Driven Development
Write code for humans first, machines second.
Yes
classShoppingCart:def__init__(self):self.items=[]defadd_item(self,item):"""Add an item to the cart."""self.items.append(item)defget_total(self):"""Calculate total price of all items."""returnsum(item.priceforiteminself.items)
Note: Leading underscores help IDEs identify protected/private members and can trigger “unused” warnings for internal helpers
Private methods
__double_leading_underscore(self, ...)
Constants
ALL_CAPS_WITH_UNDERSCORES
About underscore prefixes:
Using _ prefix for protected/private functions helps your IDE and linters understand your intent:
# mymodule.py
def_internal_helper(data):"""Internal function - not part of public API."""returndata.strip().lower()defpublic_function(text):"""Public API function."""return_internal_helper(text)# IDE will warn that _internal_helper is "unused" if not called internally
# IDE will mark it as protected/internal in code navigation
Rationale: IDEs like PyCharm and VS Code use the underscore prefix to:
Activate “unused code” warnings for internal helpers
Indicate these functions shouldn’t be imported with from module import *
Show different icons/colors in code navigation to distinguish public vs internal APIs
General Naming Guidelines
Avoid one-letter variables (esp. l, O, I).
Exception: In very short blocks, when the meaning is clearly visible from the immediate context
Avoid reusing the same variable name for different purposes.
Yes
# Each variable has a single, clear purpose
user_input=input("Enter a number: ")user_number=int(user_input)squared_number=user_number**2print(f"Result: {squared_number}")
# Processing different types of data
raw_data=fetch_data_from_api()processed_data=clean_data(raw_data)validated_data=validate_data(processed_data)
No
# 'data' means something different each time
data=input("Enter a number: ")# data is a string
data=int(data)# now data is an int
data=data**2# now data is the squared result
print(f"Result: {data}")# confusing!
# Reusing 'result' for unrelated things
result=calculate_tax(price)save_to_database(result)result=send_email(user)# result now means something completely different
result=validate_input(form)# and now something else again
Rationale: Reusing variable names makes code harder to debug, understand, and maintain. Each variable should represent one concept throughout its scope.
However, note that: A tab could be a different number of columns depending on your environment, but a space is always one column. In terms of how many spaces (or tabs) constitutes indentation, it’s more important to be consistent throughout your code than to use any specific tab stop value.
When possible, use one-line docstrings for obvious functions.
"""Return the pathname of ``foo``."""
Multiline docstrings should include
Summary line
Use case, if appropriate
Args
Return type and semantics, unless None is returned
"""Train a model to classify Foos and Bars.
Usage::
>>>importklassify>>>data=[("green","foo"),("orange","bar")]>>>classifier=klassify.train(data):param train_data: A list of tuples of the form ``(color, label)``.
:rtype: A :class:`Classifier <Classifier>`
"""
Notes
Use action words (“Return”) rather than descriptions (“Returns”).
Document __init__ methods in the docstring for the class.
classPerson(object):"""A simple representation of a human being.
:param name: A string, the person's name.
:param age: An int, the person's age.
"""def__init__(self,name,age):self.name=nameself.age=age
Alternative Docstring Formats
While the examples above use reStructuredText (reST) format, there are other popular docstring styles used in the Python community. Choose one and be consistent throughout your project.
NumPy/SciPy Style
Popular in scientific computing, more readable for complex functions with many parameters.
defcalculate_statistics(data,weights=None,ddof=0):"""
Calculate mean and standard deviation of data.
Parameters
----------
data : array_like
Input data array.
weights : array_like, optional
Weights for each value in data. Default is None.
ddof : int, optional
Delta degrees of freedom. Default is 0.
Returns
-------
mean : float
Arithmetic mean of the data.
std : float
Standard deviation of the data.
Raises
------
ValueError
If data is empty.
Examples
--------
>>>calculate_statistics([1,2,3,4,5])(3.0,1.4142135623730951)"""pass
Google Style
Clean and readable, popular in many open-source projects.
defcalculate_statistics(data,weights=None,ddof=0):"""Calculate mean and standard deviation of data.
Args:
data (array_like): Input data array.
weights (array_like, optional): Weights for each value in data.
Defaults to None.
ddof (int, optional): Delta degrees of freedom. Defaults to 0.
Returns:
tuple: A tuple containing:
- mean (float): Arithmetic mean of the data.
- std (float): Standard deviation of the data.
Raises:
ValueError: If data is empty.
Examples:
>>>calculate_statistics([1,2,3,4,5])(3.0,1.4142135623730951)"""pass
Comparison
Style
Best For
Tools
reStructuredText (reST)
General Python projects, Sphinx documentation
Sphinx, most IDEs
NumPy/SciPy
Scientific computing, data science projects
Sphinx with Napoleon extension
Google
Clean, readable docs; Google-style projects
Sphinx with Napoleon extension
Documentation Generation Tools
Beyond Sphinx, several other tools can generate documentation from your docstrings:
When you do write comments, remember: “Strunk and White apply.” - PEP 8
In summary:
Use clear, direct language and avoid unnecessary words to ensure a reader’s understanding, as recommended by Strunk and White.
Line lengths
Don’t stress over it. 80-100 characters is fine. We have wide screens nowadays.
Use parentheses for line continuations.
wiki=("The Colt Python is a .357 Magnum caliber revolver formerly manufactured ""by Colt's Manufacturing Company of Hartford, Connecticut. It is sometimes "'referred to as a "Combat Magnum". It was first introduced in 1955, the '"same year as Smith & Wesson's M29 .44 Magnum.")
String Formatting
Use f-strings (formatted string literals) for string formatting. They are more readable, concise, and faster than older methods.
Yes
name="Alice"age=30city="Lisbon"# Simple variable interpolation
message=f"Hello, {name}!"# Expressions inside f-strings
info=f"{name} is {age} years old and lives in {city}."# Formatting numbers
price=19.99quantity=3total=f"Total: €{price*quantity:.2f}"# Total: €59.97
# Multi-line f-strings
report=(f"User Report:\n"f" Name: {name}\n"f" Age: {age}\n"f" City: {city}")
No
name="Alice"age=30city="Lisbon"# Old-style % formatting (avoid)
message="Hello, %s!"%nameinfo="%s is %d years old and lives in %s."%(name,age,city)# str.format() method (verbose)
message="Hello, {}!".format(name)info="{} is {} years old and lives in {}.".format(name,age,city)# String concatenation (error-prone and hard to read)
message="Hello, "+name+"!"info=name+" is "+str(age)+" years old and lives in "+city+"."
Rationale: F-strings (Python 3.6+) are faster, more readable, and less error-prone than % formatting or .format(). They allow expressions directly inside the string and make the code’s intent clearer.
Advanced f-string features
# Debugging with f-strings (Python 3.8+)
x=10y=20print(f"{x=}, {y=}, {x+y=}")# x=10, y=20, x+y=30
# Calling functions
defget_status():return"active"status_msg=f"System is {get_status()}"
Formatting numbers with f-strings
# Format a float to 2 decimal places
price=19.98765formatted_price=f"{price:.2f}"# '19.99'
# Format an integer with leading zeros (e.g., pad to 4 digits)
order_number=42formatted_order=f"{order_number:04d}"# '0042'
# Right-align a string with spaces using f-strings
text="Python"width=12right_aligned=f"{text:>{width}}"print(f"'{right_aligned}'")# Output: ' Python'
# Or, for left alignment (for comparison):
left_aligned=f"{text:<{width}}"print(f"'{left_aligned}'")# Output: 'Python '
# Format a percentage with 1 decimal place
success_rate=0.857formatted_rate=f"{success_rate:.1%}"# '85.7%'
# Format large numbers with commas
population=1234567formatted_population=f"{population:,}"# '1,234,567'
# Format large numbers with locale-aware grouping using `:n`
importlocalelocale.setlocale(locale.LC_ALL,'')# Set to user's default locale
large_number=1234567.89formatted_number=f"{large_number:n}"# e.g., '1,234,567.89' or '1.234.567,89' depending on locale
Type Hints
Use type hints to make your code more readable and maintainable. Type hints help IDEs provide better autocomplete, catch errors early, and serve as inline documentation.
Yes
defcalculate_total(price:float,quantity:int)->float:"""Calculate total price for items."""returnprice*quantitydefgreet(name:str)->str:"""Return a greeting message."""returnf"Hello, {name}!"defprocess_items(items:list[str])->dict[str,int]:"""Count occurrences of each item."""counts={}foriteminitems:counts[item]=counts.get(item,0)+1returncounts
No
defcalculate_total(price,quantity):"""Calculate total price for items."""returnprice*quantity# What types? Will this work with all inputs?
defgreet(name):"""Return a greeting message."""returnf"Hello, {name}!"# Is name always a string?
defprocess_items(items):"""Count occurrences of each item."""counts={}foriteminitems:counts[item]=counts.get(item,0)+1returncounts# What does this return?
Rationale: Type hints improve code clarity, enable better IDE support (autocomplete, refactoring, error detection), and help catch bugs before runtime. Modern IDEs like VS Code and PyCharm use type hints to provide intelligent code completion and warnings.
Modern type hints (Python 3.9+)
Starting with Python 3.9, you can use built-in types directly instead of importing from typing. Python 3.10+ also introduces the | operator for union types.
Yes (Python 3.10+)
# Use built-in types with [] syntax (Python 3.9+)
defprocess_items(items:list[str])->dict[str,int]:"""Count occurrences of each item."""counts={}foriteminitems:counts[item]=counts.get(item,0)+1returncounts# Use | operator for unions (Python 3.10+)
deffind_user(user_id:int)->str|None:"""Find user by ID, return None if not found."""ifuser_id>0:return"Alice"returnNonedefprocess_id(id_value:int|str)->str:"""Process ID that can be int or string."""returnstr(id_value)# Multiple union types
defparse_config(value:str|int|float|None)->str:"""Parse configuration value."""returnstr(value)ifvalueisnotNoneelse""
Older style (Python 3.5-3.8)
fromtypingimportOptional,Union,List,Dict# Had to import and use capitalized generic types
defprocess_items(items:List[str])->Dict[str,int]:"""Count occurrences of each item."""counts={}foriteminitems:counts[item]=counts.get(item,0)+1returncounts# Used Optional and Union from typing module
deffind_user(user_id:int)->Optional[str]:"""Find user by ID, return None if not found."""ifuser_id>0:return"Alice"returnNonedefprocess_id(id_value:Union[int,str])->str:"""Process ID that can be int or string."""returnstr(id_value)
Note: If you’re using Python 3.10 or later, prefer the modern syntax with | and built-in types (list, dict, set, tuple). For Python 3.9, use built-in types with [] but continue using Optional and Union from typing. The old style with List, Dict, etc. from typing is now deprecated but still works.
More type hint examples
fromtypingimportAny,Callable# Class type hints
classUser:def__init__(self,name:str,age:int)->None:self.name:str=nameself.age:int=agedefget_info(self)->dict[str,Any]:"""Return user information."""return{"name":self.name,"age":self.age}# Callable type hints
defapply_operation(x:int,operation:Callable[[int],int])->int:"""Apply a function to a number."""returnoperation(x)# Type aliases for complex types
UserId=intUserData=dict[str,str|int]# Python 3.10+
# Or for older versions: dict[str, Union[str, int]]
defget_user_data(user_id:UserId)->UserData:"""Get user data by ID."""return{"name":"Alice","age":30}
IDE and AI Assistant Benefits
With type hints, your IDE and AI coding assistants can:
Autocomplete: Suggest methods and attributes based on the type
Error detection: Warn you when passing wrong types before running code
Refactoring: Safely rename variables and functions across your codebase
Documentation: Show parameter types in function signatures without reading docs
AI assistance: GitHub Copilot, Codeium, and other AI assistants provide more accurate suggestions when they understand your types
# IDE knows 'result' is a float, suggests float methods
result:float=calculate_total(19.99,3)result.is_integer()# IDE autocompletes this method
# IDE warns if you pass wrong types
calculate_total("19.99","3")# IDE shows warning: expected float and int
# AI assistants provide better completions with type hints
defprocess_users(users:list[dict[str,str]])->list[str]:# AI knows 'users' is a list of dicts, suggests appropriate operations
# AI knows return type should be list of strings
return[user["name"]foruserinusers]# AI suggests this correctly
Note: AI coding assistants like GitHub Copilot use type hints to understand your code’s intent and provide more accurate, context-aware suggestions. Well-typed code gets better AI assistance.
Leverage the Standard Library
Python’s standard library is extensive and well-tested. Before writing custom solutions or installing third-party packages, check if the standard library already provides what you need. Familiarizing yourself with common standard library modules will make you a more effective Python programmer.
collections - Specialized Container Data Types
The collections module provides alternatives to built-in containers with additional functionality.
Counter - Count hashable objects
Yes
fromcollectionsimportCounter# Count occurrences in a list
words=["apple","banana","apple","orange","banana","apple"]word_counts=Counter(words)print(word_counts)# Counter({'apple': 3, 'banana': 2, 'orange': 1})
# Get most common items
print(word_counts.most_common(2))# [('apple', 3), ('banana', 2)]
# Combine counters
more_words=["apple","grape"]word_counts.update(more_words)
No
# Manually counting (verbose and error-prone)
words=["apple","banana","apple","orange","banana","apple"]word_counts={}forwordinwords:ifwordinword_counts:word_counts[word]+=1else:word_counts[word]=1
defaultdict - Dictionary with default values
fromcollectionsimportdefaultdict# Group items by category
items=[("fruit","apple"),("fruit","banana"),("vegetable","carrot")]grouped=defaultdict(list)forcategory,iteminitems:grouped[category].append(item)# {'fruit': ['apple', 'banana'], 'vegetable': ['carrot']}
functools - Higher-Order Functions
partial - Partial function application
Yes
fromfunctoolsimportpartialdefpower(base:int,exponent:int)->int:"""Raise base to the power of exponent."""returnbase**exponent# Create specialized functions
square=partial(power,exponent=2)cube=partial(power,exponent=3)print(square(5))# 25
print(cube(5))# 125
No
# Creating wrapper functions manually
defpower(base:int,exponent:int)->int:"""Raise base to the power of exponent."""returnbase**exponentdefsquare(base:int)->int:returnpower(base,2)defcube(base:int)->int:returnpower(base,3)
lru_cache - Memoization decorator
fromfunctoolsimportlru_cache@lru_cache(maxsize=128)deffibonacci(n:int)->int:"""Calculate nth Fibonacci number with caching."""ifn<2:returnnreturnfibonacci(n-1)+fibonacci(n-2)# Much faster for repeated calls
print(fibonacci(100))# Computed instantly due to caching
itertools - Iterator Building Blocks
chain - Combine multiple iterables
fromitertoolsimportchain# Combine multiple lists into one
list1=[1,2,3]list2=[4,5,6]combined=list(chain(list1,list2))# [1, 2, 3, 4, 5, 6]
combinations - All combinations of items
fromitertoolsimportcombinations# Get all 2-item combinations
items=['A','B','C']combos=list(combinations(items,2))# [('A', 'B'), ('A', 'C'), ('B', 'C')]
fromitertoolsimportgroupby# Group consecutive items by a key
data=[('A',1),('A',2),('B',1),('B',2),('A',3)]forkey,groupingroupby(data,key=lambdax:x[0]):print(f"{key}: {list(group)}")# A: [('A', 1), ('A', 2)]
# B: [('B', 1), ('B', 2)]
# A: [('A', 3)]
islice - Slice an iterator
fromitertoolsimportislice# Get items from an iterator without loading all into memory
defgenerate_numbers():n=0whileTrue:yieldnn+=1# Get first 10 even numbers
evens=(xforxingenerate_numbers()ifx%2==0)first_ten_evens=list(islice(evens,10))# [0, 2, 4, 6, 8, 10, 12, 14, 16, 18]
pathlib - Object-Oriented File Paths
Yes
frompathlibimportPath# Modern, readable path operations
project_dir=Path("/home/user/project")config_file=project_dir/"config"/"settings.json"ifconfig_file.exists():content=config_file.read_text()# List all Python files
python_files=list(project_dir.glob("**/*.py"))
Rationale: The standard library is well-documented, thoroughly tested, and requires no additional dependencies. Learning these modules will help you write more concise, efficient, and idiomatic Python code.
Modern Python Features
Python continues to evolve with each version, introducing new syntax and features that make code more readable and concise. Here are some significant additions from recent Python versions.
The walrus operator := allows you to assign values to variables as part of an expression. This is particularly useful in conditions and loops.
Yes
# Assign and use in one line
if (n:=len(data))>10:print(f"List is too long ({n} elements, expected <= 10)")# Avoid redundant function calls in while loops
while (line:=file.readline())!="":process(line)# List comprehensions with shared subexpression
data=[1,2,3,4,5]results=[yforxindataif (y:=x*2)>5]# [6, 8, 10]
# Reuse expensive computation in comprehension
expensive_results=[resultforxindataif (result:=expensive_function(x))isnotNone]
No
# Without walrus operator - less concise
n=len(data)ifn>10:print(f"List is too long ({n} elements, expected <= 10)")# Redundant calls
line=file.readline()whileline!="":process(line)line=file.readline()# Computing twice
results=[x*2forxindataifx*2>5]
Rationale: The walrus operator reduces code duplication and makes intentions clearer, especially when you need to use a computed value multiple times.
Positional-Only and Keyword-Only Parameters - Python 3.8+
Use / to specify positional-only parameters and * for keyword-only parameters, making your API more explicit and preventing misuse.
Yes
# Positional-only parameters (before /)
defgreet(name,/,greeting="Hello"):"""Name must be positional, greeting can be keyword."""returnf"{greeting}, {name}!"greet("Alice")# OK
greet("Alice",greeting="Hi")# OK
# greet(name="Alice") # Error: name is positional-only
# Keyword-only parameters (after *)
defcreate_user(username,*,email,age):"""Email and age must be passed as keywords."""return{"username":username,"email":email,"age":age}create_user("alice",email="alice@example.com",age=30)# OK
# create_user("alice", "alice@example.com", 30) # Error: email and age must be keyword
# Combining both
defprocess(a,b,/,c,*,d,e):"""a, b are positional-only; c can be either; d, e are keyword-only."""pass
No
# Ambiguous API without parameter restrictions
defgreet(name,greeting="Hello"):returnf"{greeting}, {name}!"# Users might do this (unclear which is which):
greet(greeting="Alice",name="Hello")# Confusing!
Rationale: Explicit parameter types prevent API misuse and make function signatures self-documenting.
The match/case statement provides powerful pattern matching similar to switch statements in other languages, but much more flexible.
Yes
# Simple value matching
defhttp_status(status:int)->str:matchstatus:case200:return"OK"case404:return"Not Found"case500:return"Internal Server Error"case_:return"Unknown Status"# Pattern matching with structures
defprocess_command(command:tuple)->str:matchcommand:case ("quit",):return"Exiting..."case ("load",filename):returnf"Loading {filename}"case ("save",filename,format):returnf"Saving {filename} as {format}"case_:return"Unknown command"# Matching objects and types
defdescribe(obj):matchobj:caseint(x)ifx>0:returnf"Positive integer: {x}"caseint(x)ifx<0:returnf"Negative integer: {x}"casestr(s)iflen(s)>0:returnf"Non-empty string: {s}"caselist([]):return"Empty list"caselist([first,*rest]):returnf"List starting with {first}"case{"name":name,"age":age}:returnf"Person: {name}, age {age}"case_:return"Something else"
No
# Long if-elif chains (less readable)
defhttp_status(status:int)->str:ifstatus==200:return"OK"elifstatus==404:return"Not Found"elifstatus==500:return"Internal Server Error"else:return"Unknown Status"defprocess_command(command:tuple)->str:iflen(command)==1andcommand[0]=="quit":return"Exiting..."eliflen(command)==2andcommand[0]=="load":returnf"Loading {command[1]}"eliflen(command)==3andcommand[0]=="save":returnf"Saving {command[1]} as {command[2]}"else:return"Unknown command"
Rationale: Pattern matching makes complex conditional logic more readable and maintainable, especially when dealing with structured data.
Dictionary Merge and Update Operators - Python 3.9+
Use | and |= operators for merging dictionaries.
Yes
# Merge dictionaries with | operator
defaults={"color":"blue","size":"medium"}custom={"size":"large","style":"modern"}config=defaults|custom# {'color': 'blue', 'size': 'large', 'style': 'modern'}
# Update in place with |=
settings={"theme":"dark"}settings|={"language":"en","theme":"light"}# settings is now {'theme': 'light', 'language': 'en'}
No
# Old way - more verbose
defaults={"color":"blue","size":"medium"}custom={"size":"large","style":"modern"}config={**defaults,**custom}# Works but less clear
# Or even older
config=defaults.copy()config.update(custom)
Rationale: The | operator makes dictionary merging more intuitive and consistent with set operations.
Improved Error Messages - Python 3.10+
Python 3.10+ provides much more helpful error messages with better context and suggestions.
# Example: Better syntax error messages
# Python 3.10+ will point to the exact location and suggest fixes
# Missing colon
# if x > 5
# ^
# SyntaxError: expected ':'
# Better name error suggestions
name="Alice"# print(nam)
# NameError: name 'nam' is not defined. Did you mean: 'name'?
# Better attribute errors
classUser:def__init__(self):self.username="alice"user=User()# print(user.usrname)
# AttributeError: 'User' object has no attribute 'usrname'. Did you mean: 'username'?
Parenthesized Context Managers - Python 3.10+
Write cleaner code when using multiple context managers.
Yes (Python 3.10+)
# Parentheses allow natural line breaks
with (open("input.txt")asinfile,open("output.txt","w")asoutfile,open("log.txt","w")aslogfile,):process_files(infile,outfile,logfile)
Rationale: Parenthesized context managers improve readability when working with multiple resources.
Essential Python Idiomatics and Best Practices
The if __name__ == "__main__" Pattern
Use this idiomatic expression to make your Python files both importable modules and executable scripts.
Yes
# mymodule.py
defcalculate_sum(a:int,b:int)->int:"""Add two numbers."""returna+bdefmain():"""Main entry point when run as script."""result=calculate_sum(5,3)print(f"Result: {result}")if__name__=="__main__":main()
When imported:
# other_file.py
importmymodule# Only the function is available, main() doesn't run automatically
result=mymodule.calculate_sum(10,20)
When run directly:
python mymodule.py # Runs main() and prints "Result: 8"
No
# mymodule.py - Bad practice
defcalculate_sum(a:int,b:int)->int:"""Add two numbers."""returna+b# Code runs immediately when imported (bad!)
result=calculate_sum(5,3)print(f"Result: {result}")
Rationale: This pattern allows code reuse. The same file can be used as an importable module or run as a standalone script without executing unwanted side effects during import.
Logging vs Print
Use the logging module instead of print() for anything beyond simple scripts. Logging provides better control over output levels, formatting, and destinations.
Yes
importlogging# Configure logging at the start of your application
logging.basicConfig(level=logging.INFO,format='%(asctime)s - %(name)s - %(levelname)s - %(message)s')logger=logging.getLogger(__name__)defprocess_data(data):logger.debug(f"Processing data: {data}")ifnotdata:logger.warning("Empty data received")returnNonetry:result=expensive_operation(data)logger.info(f"Successfully processed {len(data)} items")returnresultexceptExceptionase:logger.error(f"Failed to process data: {e}",exc_info=True)raise# Benefits:
# - Can change log level without modifying code
# - Logs include timestamps and context
# - Can redirect to files, streams, or external services
# - Different loggers for different modules
No
defprocess_data(data):print(f"Processing data: {data}")# Always prints, no control
ifnotdata:print("WARNING: Empty data received")# No standard format
returnNonetry:result=expensive_operation(data)print(f"Processed {len(data)} items")# Mixed with actual output
returnresultexceptExceptionase:print(f"ERROR: {e}")# No stack trace, hard to debug
raise
Log Levels
importlogginglogger=logging.getLogger(__name__)# Use appropriate levels
logger.debug("Detailed information for debugging")logger.info("General informational messages")logger.warning("Warning messages for unexpected situations")logger.error("Error messages for serious problems")logger.critical("Critical messages for very serious errors")# In production, set level to INFO or WARNING to reduce noise
logging.basicConfig(level=logging.WARNING)# Now only warning, error, and critical messages appear
Rationale: Logging is configurable, structured, and production-ready. It allows you to control verbosity without changing code and provides better context for debugging.
Efficient Collection Operations
Choose the right data structure for the job. Understanding performance characteristics prevents inefficient code. For more use cases, refer to this blog post.
Set Membership Testing
Yes
# Use sets for membership testing - O(1) average case
valid_users={"alice","bob","charlie","david"}# set
ifusernameinvalid_users:# Fast: O(1)
grant_access()# Converting list to set for multiple lookups
user_list=["alice","bob","charlie","david"]user_set=set(user_list)# Convert once
forlogin_attemptinlogin_attempts:iflogin_attemptinuser_set:# O(1) per check
process_login(login_attempt)
No
# Using lists for membership testing - O(n) linear search
valid_users=["alice","bob","charlie","david"]# list
ifusernameinvalid_users:# Slow: O(n)
grant_access()# Repeatedly searching through lists
forlogin_attemptinlogin_attempts:iflogin_attemptinvalid_users:# O(n) per check - very slow!
process_login(login_attempt)
Dictionary for Fast Lookups
Yes
# Use dicts for key-value lookups - O(1)
user_scores={"alice":95,"bob":87,"charlie":92}score=user_scores.get("alice",0)# Fast lookup with default
# Dict comprehension
squared={x:x**2forxinrange(10)}# {0: 0, 1: 1, 2: 4, 3: 9, ...}
No
# Using parallel lists - inefficient and error-prone
usernames=["alice","bob","charlie"]scores=[95,87,92]# Linear search to find score - O(n)
defget_score(username):fori,nameinenumerate(usernames):ifname==username:returnscores[i]return0
Set Operations
# Set operations are fast and readable
set_a={1,2,3,4,5}set_b={4,5,6,7,8}# Union - all unique elements
union=set_a|set_b# {1, 2, 3, 4, 5, 6, 7, 8}
# Intersection - common elements
intersection=set_a&set_b# {4, 5}
# Difference - elements in a but not in b
difference=set_a-set_b# {1, 2, 3}
# Symmetric difference - elements in either but not both
sym_diff=set_a^set_b# {1, 2, 3, 6, 7, 8}
# Remove duplicates from a list
unique_items=list(set(items_with_duplicates))
List vs Generator for Large Data
Yes
# Generator - memory efficient for large data
defread_large_file(filename):withopen(filename)asf:forlineinf:# Processes one line at a time
yieldline.strip()# Only loads one line at a time
forlineinread_large_file("huge_file.txt"):process(line)# Generator expression
sum_of_squares=sum(x**2forxinrange(1000000))# Memory efficient
No
# List - loads everything into memory
defread_large_file(filename):withopen(filename)asf:return[line.strip()forlineinf]# Loads entire file!
# Memory intensive
all_lines=read_large_file("huge_file.txt")forlineinall_lines:process(line)# List comprehension - creates full list in memory
sum_of_squares=sum([x**2forxinrange(1000000)])# Wasteful
Rationale: Using the right data structure improves performance dramatically. Sets and dicts offer O(1) lookups, while lists require O(n) searches. Generators save memory for large datasets.
Duck Typing and EAFP vs LBYL
Duck typing is a programming concept where the type or class of an object is determined by its behavior (methods and properties), rather than its explicit inheritance or type. The phrase “If it walks like a duck and quacks like a duck, it’s a duck” means that if an object implements the necessary methods or behaviors, it can be used wherever those behaviors are expected—regardless of its actual type. This approach makes code more flexible and reusable, as it focuses on what an object can do, not what it is. Python also follows the philosophy of “Easier to Ask for Forgiveness than Permission” (EAFP) rather than “Look Before You Leap” (LBYL).
EAFP - Pythonic Approach
Yes
# EAFP: Try the operation, handle exceptions if it fails
defget_user_age(user_dict):try:returnuser_dict["age"]exceptKeyError:returnNone# Duck typing: "If it walks like a duck and quacks like a duck..."
defprocess_file(file_obj):# Don't check type - just use it like a file
try:content=file_obj.read()returncontent.upper()exceptAttributeError:raiseTypeError("Object must have a read() method")# Works with any file-like object
fromioimportStringIOprocess_file(open("file.txt"))# Real file
process_file(StringIO("test"))# String buffer - also works!
# EAFP with dict access
config={"timeout":30,"retries":3}try:timeout=config["timeout"]retries=config["retries"]exceptKeyErrorase:raiseValueError(f"Missing required config: {e}")
No - LBYL (Look Before You Leap)
# LBYL: Check before you try (less Pythonic, race conditions)
defget_user_age(user_dict):if"age"inuser_dict:# Extra check
returnuser_dict["age"]else:returnNone# Type checking instead of duck typing (rigid)
defprocess_file(file_obj):ifnotisinstance(file_obj,io.IOBase):# Too restrictive!
raiseTypeError("Must be a file object")content=file_obj.read()returncontent.upper()# Now StringIO won't work, even though it has read()!
# Multiple checks (verbose and slower)
config={"timeout":30,"retries":3}if"timeout"inconfigand"retries"inconfig:timeout=config["timeout"]retries=config["retries"]else:raiseValueError("Missing required config")
More EAFP Examples
# Converting to int
# EAFP - Pythonic
try:value=int(user_input)exceptValueError:print("Invalid number")# LBYL - Not Pythonic
ifuser_input.isdigit():value=int(user_input)else:print("Invalid number")# Problem: isdigit() doesn't handle negative numbers or floats
# File operations
# EAFP - Pythonic
try:withopen("config.json")asf:config=json.load(f)exceptFileNotFoundError:config=default_config()exceptjson.JSONDecodeError:print("Invalid JSON")# LBYL - Less Pythonic
importosifos.path.exists("config.json"):# Race condition: file could be deleted between check and open!
withopen("config.json")asf:config=json.load(f)else:config=default_config()
Duck Typing Benefits
# Functions work with any compatible type
defprint_all(items):"""Works with lists, tuples, sets, generators, etc."""foriteminitems:# Just needs to be iterable
print(item)print_all([1,2,3])# list
print_all((1,2,3))# tuple
print_all({1,2,3})# set
print_all(range(1,4))# range object
print_all(xforxin[1,2,3])# generator
# Custom class works too if it implements __iter__
classMyCollection:def__init__(self,data):self.data=datadef__iter__(self):returniter(self.data)print_all(MyCollection([1,2,3]))# Works!
Rationale: EAFP is more Pythonic, often faster (one operation vs check + operation), and handles edge cases better. Duck typing makes code more flexible and reusable by focusing on behavior rather than type.
]]>Analyzing the history of CVPR2025-04-25T16:43:13+00:002025-04-25T16:43:13+00:00https://george-gca.github.io/blog/2025/cvpr-history-analysisAs the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2025 approaches, let’s take a look at the history of the conference and its workshops from 2017 to 2024. The goal of this analysis is to provide insights into the evolution of topics and trends in artificial intelligence research over the years. Keep in mind that this information should be taken with a grain of salt, as some of the information that may be relevant to the analyses is discarded during the cleaning process. Some of the analysis is based on keywords, and we make some assumptions about how authors use keywords (e.g. it’s pretty unlikely that a paper about image data would have the keyword audio in its title or abstract), but this is not a perfect solution. The goal of this post is to give some insight into the history of the conference, not to be a definitive analysis.
Note that some of the graphs use percentiles of the total number of papers published in each year. Since there are different numbers of papers published each year, you can’t really compare the numbers from one year to the next. The goal of these graphs is to show the distribution of papers published during the period and any changes in the focus of the academic community. You can also interact with the visualizations here. You can zoom in on specific parts, enable or disable lines by clicking on their names in the legend, and hover over the points to see more information.
Overall Statistics
Here you can see the number of published papers. Each year, there are more and more papers published compared to the previous year, except for 2023. There were more than three times as many papers published in 2024 as in 2017.
Regarding the modalities used in the papers, we can see that image is still the most common, but the use of text and multiple modalities has increased significantly. The application of optical flow, graphs and depth information has decreased in the last years, while the use of particles has remained relatively stable.
It is quite common for papers to introduce new concepts, be it a new method, a new dataset, or a new architecture. The following graph shows the most common concepts introduced in the papers. Not surprisingly, algorithms are the most common concept. Algorithms also involve new methods or approaches. Novel tasks have also been introduced over the years, which is highly correlated with the creation of novel datasets. The introduction of new architectures has also increased in the last year, including new models, modules, and networks. The creation of different losses and metrics has been quite stable over the years, with very few papers introducing new ones.
Regarding the common tasks in the papers, we can see a steep increase in generation tasks, especially after 2022. This may be related to the advances in large language models such as InstructGPT and ChatGPT by the end of 2022, and the release of the first collections of foundational language models such as LLaMA in early 2023. Classification, detection, estimation, and recognition have seen a decline in interest over the years, while prediction has only recently seen a decrease. Tasks such as segmentation have remained relatively stable. The use of reasoning tasks has also increased significantly in the last year, but is still a small percentage of the total number of published papers (about 3%).
Algorithms focused on security and privacy have been around for a while, but the number of papers published on them has increased significantly in the last year. Spoofing detection is crucial for applications such as identity recognition, where attackers may try to use photos or videos to impersonate someone else, and has seemed to gain urgency since deepfake technologies have become more prevalent.
Explainability and interpretability has gained traction in the last few years, with a significant increase in the number of papers published on the topic around 2019, following a surge in some specific conferences and workshops on model transparency, interpretability, and fairness, such as ACM FaccT and VISxAI. Explainability is crucial for building trust in AI systems and ensuring that they make decisions based on valid reasoning. One of the areas that has seen the most investment in recent years is model grounding, the process of tying the model’s predictions to specific features in the input data. This is particularly important in applications such as image classification and question answering, where it is essential to understand which parts of an input (text, image) are driving the model’s predictions.
Visual tasks such as image denoising have received a lot of attention in recent years, with many papers published on the topic. This may be due to the increasing importance of image quality in computer vision applications, the development of new techniques to improve image quality, and the increased capacity of visual models to handle larger inputs. This category of tasks also includes deblurring, dehazing, demoireing, deraining, and others. Image processing and image generation tasks have also increased significantly.
Language tasks have also seen a fluctuation in the number of papers published over the past few years, particularly those that focus on dialogue and conversation. By using a conversational interface, users can interact with AI systems in a more natural and intuitive way, leading to better user experiences and more effective communication. This has led to a surge in research on dialog systems, including chatbots, virtual assistants, and other conversational agents. The development of large-scale language models has also played a significant role in this trend, as these models have demonstrated impressive capabilities in generating human-like text and understanding context.
Multimodal tasks are one of the current trends in artificial intelligence. These tasks involve the combination of different modalities, such as audio, text, and images, to improve the performance of models and to solve problems that require a deeper understanding of the intermodality of the world. The number of papers published on these tasks has increased significantly in recent years, with a particular focus on tasks such as image-text alignment, image synthesis, video synthesis, and visual question answering. This trend is likely to continue as researchers explore new ways to combine different modalities in novel ways and improve the performance of models.
Here we focus on analyzing the use of some keywords in the LLM papers. More specifically:
Chain-of-Thought, Tree-of-Thought, and any of-Thought variations - these are prompting techniques that help the model to break down complex tasks into smaller, more manageable steps, allowing it to reason through the problem more effectively;
Agent - refers to the use of LLMs as agents that can perform tasks autonomously, often in conjunction with other tools or systems;
Distillation - a technique used to compress large models into smaller, more efficient ones while retaining their performance;
Few-shot prompting - a prompting technique that provides the model with a few examples of the task at hand, allowing it to generalize and perform well on similar tasks;
Fine-tuning - the process of training a pre-trained model on a specific task or dataset to improve its performance;
Reinforcement Learning (RL) - a type of machine learning where an agent learns to make decisions by receiving feedback from its environment in the form of rewards or penalties;
Retrieval Augmented Generation (RAG) - a technique that combines retrieval-based methods with generative models to improve the performance of language models on specific tasks;
Self-Instruct - a technique that allows models to learn from their own outputs, improving their performance over time;
Tokenizer - a component of language models that converts text into a format that the model can understand, often by breaking it down into smaller units called tokens;
Tool - refers to the use of external tools or systems in conjunction with LLMs to perform tasks more effectively;
Zero-shot prompting - a prompting technique that allows the model to perform tasks without any prior examples or training on that specific task.
Few-shot and zero-shot prompting have lost the interest of the academic community in favor of RAG, thought processes, and novel fine-tuning techniques. Interest in creating LLM agents that can tackle harder tasks and use tools is one of the hottest topics in the field.
Now let’s look at the authors of the papers. This first graph shows the number of papers published by each author. As we can see, most authors have published only one paper at the conference. Out of 33,861 authors, only 1,308 have 10 or more accepted papers.
Now let’s look at the number of authors per paper. Most of the papers have between 2 and 7 authors, but there are a few with a large number of authors, such as Why Is the Winner the Best?, which has 125 authors, and The Ninth NTIRE 2024 Efficient Super-Resolution Challenge Report, with a staggering 134 authors. The former is a multi-center study of all 80 competitions held as part of IEEE ISBI 2021 and MICCAI 2021, while the latter is a report summarizing the results of the NTIRE 2024 challenge, a competition held at the CVPR conference.
Since most papers have multiple authors, it is quite common to see some authors constantly collaborating with each other. The most common pair of authors is Jiwen Lu and Jie Zhou, who have collaborated on 57 papers together. The second most common pair is Luc Van Gool and Radu Timofte with 43 papers together, followed by Tao Xiang and Yi-Zhe Song with 38 papers. The top 10 most frequent pairs of authors are:
Author 1
Author 2
Papers
Jiwen Lu
Jie Zhou
57
Luc Van Gool
Radu Timofte
43
Tao Xiang
Yi-Zhe Song
38
Fahad Shahbaz Khan
Salman Khan
33
Ting Yao
Tao Mei
32
Xiaogang Wang
Hongsheng Li
28
Shiguang Shan
Xilin Chen
27
Richa Singh
Mayank Vatsa
26
Dong Chen
Fang Wen
24
Yi-Zhe Song
Ayan Kumar Bhunia
24
Although it is quite rare for a paper to have a single author, 185 papers fall into this category. A few worthy mentions are research that introduced novel loss functions (Jonathan T. Barron, Takumi Kobayashi) and improved transformer architectures and post-training techniques (Takumi Kobayashi, Jing Ma). In this table we can see the authors with the most papers where they are the only author:
Author
Papers
Takumi Kobayashi
4
Anant Khandelwal, Takuhiro Kaneko
3
Andrey V. Savchenko, Chong Yu, Dimitrios Kollias, Edgar A. Bernal, Jamie Hayes, Magnus Oskarsson, Ming Li, Oleksii Sidorov, Ren Yang, Rowel Atienza, Sanghwa Hong, Satoshi Ikehata, Shunta Maeda, Stamatios Lefkimmiatis, Ying Zhao
2
Identifying Topics
For this section, we used Top2Vec, an automatic topic modeling algorithm, to identify groups of papers that are similar to each other based on their titles and abstracts. The solution found 172 topics, which is a bit too many for us to analyze individually. Instead, we will focus on the hottest and coldest topics, which are those with the most and least papers in the last year, respectively.
One problem with the algorithm is that it identifies topics based on the words used in the papers, but it doesn’t provide a clear explanation of what those topics are about. This is a common problem with topic modeling algorithms, as they often produce results that are difficult to interpret. However, we can use LLMs to help us understand the meaning of these topics. We will use the most representative words of each topic (the words that appear most often in the papers of that topic) to generate a title and a paragraph summarizing it.
The topics below are listed in the order in which they had the most published articles last year.
Topic 1 - Instruction-Tuned Multimodal LLMs for Vision-Language Understanding (157 documents) Recent advancements in large language models (LLMs) and multimodal large language models (MLLMs) have led to remarkable capabilities in integrating visual and textual information for tasks like question answering, dialogue, and reasoning. By leveraging instruction tuning and visual prompt techniques, these models — such as vision-language models (VLMs) like CLIP — have significantly improved in-context comprehension and instruction following across modalities. Despite these achievements, challenges like hallucinations and limited applicability in complex, real-world settings still remain. Research continues to focus on enhancing multimodal instruction learning to facilitate deeper, more reliable understanding between vision and language inputs.
Topic 2 - Controllable Text-Guided Image Editing with Diffusion and GAN Inversion (426 documents) Advances in text-to-image diffusion models and GAN-based techniques have unlocked powerful, controllable image editing capabilities driven by natural language prompts. Methods like StyleGAN inversion, DDIM inversion, and textual inversion allow users to manipulate generated images or real inputs with high fidelity while preserving key features like identity. Text-guided editing leverages the latent space of pretrained diffusion and GAN models, enabling creative, precise, and personalized edits through simple prompts. Despite remarkable progress, achieving fine-grained control over complex edits and extending these capabilities to video editing remain active research challenges.
Topic 3 - AI-Driven Medical Imaging and Diagnosis in Clinical Practice (345 documents) The integration of computational methods into medical imaging and pathology has become increasingly important for improving diagnostic accuracy and early disease detection. Techniques like computer-aided diagnosis support clinicians in analyzing tissues, tumors, and organs across modalities such as histopathology, MRI, CT, and digital microscopy. Applications span cancer diagnosis (e.g., breast, brain, skin lesions), neurological diseases like Alzheimer's and Parkinson's, and blood analysis. By enhancing image analysis and tissue classification, these AI-driven tools aid in treatment planning and disease progression monitoring, making them vital in modern clinical practice and biomedical research.
Topic 4 - Challenges and Advances in 3D-Aware Text-to-Image and Text-to-Video Generation (68 documents) Text-to-image and text-to-video generation using diffusion models has made remarkable progress, enabling the synthesis of high-fidelity, photorealistic assets from simple prompts. However, existing methods still struggle with accurately handling 3D geometry, novel views, and maintaining global and multi-view consistency. Techniques like diffusion priors, 3D Gaussians, and NeRF-based approaches aim to improve subject-driven generation and diverse, globally consistent outputs. Despite advances in pretrained diffusion models and anisotropic diffusion strategies, achieving high-fidelity, geometry-aware synthesis remains a central challenge in the evolution of text-to-3D and motion generation.
Topic 5 - Remote Sensing and Aerial Imagery for Environmental and Agricultural Monitoring (306 documents) The rapid development of satellite and unmanned aerial vehicle (UAV) technologies has fueled increased interest in using high-resolution imagery for environmental, agricultural, and urban management. Remote sensing enables the monitoring of plant species, crop types, water resources, land cover changes, and urban infrastructure such as roads and buildings, particularly aiding developing countries. Applications range from crop management and plant phenotyping to traffic management and tracking environmental factors and changes. Publicly accessible satellite and aerial datasets are becoming vital tools for tackling global challenges in resource management, environmental protection, and urban planning.
Topic 6 - Competitions and Challenges in Computer Vision: The Role of NTIRE and Beyond (90 documents) Large-scale competitions like the NTIRE Challenge, MegaFace Challenge, and ABAW Competition have become central to advancing computer vision research. Hosted at major conferences like CVPR, these challenges attract hundreds of registered participants and teams, competing across various tracks such as perceptual quality, AI-generated content, and traffic analysis. Through rigorous submissions and evaluations on standardized test sets, these challenges foster innovation, benchmark progress, and tackle formidable problems in the field. The NTIRE Workshop, in particular, has established itself as a premier platform for recognizing outstanding achievements and setting new frontiers in computer vision competitions.
Topic 7 - Enhancing Diffusion Models: Faster Inference and Higher Image Quality (64 documents) Diffusion models have emerged as powerful tools for generating high-quality images, particularly in text-to-image tasks, but they often suffer from slow inference speeds and inherent limitations tied to their timestep-based denoising process. Recent advances focus on accelerating inference and improving FID scores through innovations like post-training quality enhancement, tailored token mixing (e.g., super tokens, OCR tokens), and anisotropic diffusion strategies. These methods can be flexibly applied with negligible computational overhead, substantially improving image quality without retraining. By addressing inherent inefficiencies and showcasing superior performance, these techniques represent a major step forward in diffusion-based image generation.
Topic 8 - Intelligent Traffic Monitoring and Driver Behavior Analysis for Road Safety (58 documents) Advances in intelligent transportation systems are increasingly focused on improving road safety through the analysis of driver behavior, traffic scenarios, and vehicle-pedestrian interactions. By leveraging traffic monitoring, naturalistic driving datasets, and leaderboard-driven challenges, researchers aim to develop better driver assistance systems and accident prevention technologies. Areas like distracted driver detection, traffic surveillance, and automated driving benefit from smart monitoring systems that enhance safe driving practices and reduce traffic accidents. As public leaderboards rank the best published methods, innovations in vehicle tracking, intelligent traffic analysis, and safe transportation continue to accelerate progress toward safer roads.
Topic 9 - Soccer and Sports Video Analytics: Player Tracking and Game Understanding (103 documents) Advances in sports video analytics, particularly for soccer, focus on tracking players, analyzing game states, and generating highlights from broadcast footage and sport-specific datasets. Systems capable of detecting player positions, ball movements, and team dynamics have become fundamental tools for both game analysis and automated content production. Publicly released datasets like UCF and HMDB, along with open-source code, drive innovation in this field, enabling teams around the world to develop systems capable of capturing, processing, and understanding complex sport scenarios. These developments are reshaping sports analytics, enhancing performance evaluation, and enriching fan experiences.
Topic 10 - Event-Based Vision: High-Speed, Low-Latency Sensing with Neuromorphic Cameras (129 documents) Event-based vision, powered by bio-inspired neuromorphic cameras, represents a major shift from traditional frame-based imaging. Unlike conventional sensors, event cameras capture asynchronous changes in brightness with low latency, low power consumption, and exceptional dynamic range, making them ideal for high-speed motion scenarios and environments prone to motion blur. This technology, including event-based vision for video frame interpolation (VFI) and eye tracking, has progressed rapidly, offering advantages in bandwidth efficiency and noise reduction. Applications span robotics, autonomous driving, and high-speed tracking, where conventional frame-based approaches often fall short.
The topics below are listed in the order in which they had the largest decrease in papers over the last year.
Topic 1 - Self-Supervised Pretraining: Masked Models and Their Impact on Downstream Vision Tasks (115 documents) Self-supervised pretraining has revolutionized machine learning by enabling models to learn from unlabeled data, significantly improving performance on a wide range of downstream vision tasks. Techniques like masked autoencoding and contrastive pretraining have become central to this approach, where a model is trained to predict missing parts of the data, learning rich representations without the need for labeled examples. These methods, including masked token strategies and large-scale pretraining on unlabeled video or image datasets, have shown to outperform traditional supervised pretraining, achieving great success across various applications. The benefits of self-supervised learning, especially in terms of scalability and performance, are now being extensively explored in vision-language pretraining (VLP) and other vision tasks, often surpassing existing supervised methods.
Topic 2 - Vision-Language Models: Aligning Text and Image for Cross-Modal Understanding (432 documents) Vision-language models, such as CLIP, leverage large datasets of paired image-text data to learn the alignment between visual content and language. These models enable tasks like image captioning, where a description is generated for an image, and text-based image retrieval, where a sentence or phrase is used to find relevant visual content. Through pretraining on vast collections of images and their corresponding captions, these models achieve zero-shot capabilities, meaning they can generalize to tasks they were not explicitly trained on. Grounding text in visual concepts, such as matching sentences to images or videos, has become a central challenge in creating more sophisticated systems for understanding and generating visual and textual information across diverse domains, including untrimmed videos and spoken language.
Topic 3 - Semi-Supervised Learning: Leveraging Unlabeled Data for Improved Model Performance (179 documents) Semi-supervised learning (SSL) is a powerful technique that utilizes both labeled and unlabeled data to train models, especially when labeled data is scarce. In SSL, pseudo-labeling is commonly employed, where unlabeled examples are assigned pseudo-labels based on model predictions, and these pseudo-labeled data are incorporated into the training process. This approach allows models to learn from a large amount of unlabeled data, improving generalization without requiring extensive labeled datasets. Methods like pseudo label refinement and self-training help ensure the quality and reliability of the pseudo-labels, making SSL effective for tasks like medical image analysis, where labeled data is limited. By combining labeled data with confident pseudo-labels from unlabeled examples, semi-supervised learning can outperform traditional fully supervised methods, particularly in challenging settings with partially labeled data.
Topic 4 - Domain Adaptation: Bridging the Gap Between Source and Target Domains (323 documents) Domain adaptation (DA) focuses on adapting models trained on a labeled source domain to perform well on an unseen target domain, addressing the challenges posed by domain shift or domain gap. This process is crucial when the source and target domains differ significantly, such as in cross-domain generalization tasks. Techniques like pseudo-labeling, self-training, and few-shot learning are employed to improve performance on target data, even when labeled data from the target domain is limited or unavailable. Domain adaptation methods aim to reduce the discrepancy between the source and target domains by minimizing the impact of domain shift and leveraging unlabeled target samples. These methods are vital for applications like visual domain adaptation, where new target domains with varying conditions or classes are frequently encountered.
Topic 5 - 3D Object Detection: Advancements in Lidar and Monocular Approaches for Autonomous Vehicles (217 documents) 3D object detection is a critical component of autonomous driving, enabling vehicles to perceive and understand their environment in three dimensions. Using technologies like lidar, monocular cameras, and radar, 3D detection systems create detailed representations of the surroundings, often represented in formats such as birds-eye view (BEV) or voxel grids. Datasets like KITTI, NuScenes, and Waymo provide benchmarks for evaluating 3D detection models, with lidar-based point clouds playing a central role in high-accuracy detection of objects, such as pedestrians, vehicles, and obstacles. These systems face challenges such as slow inference speeds and the complexity of predicting occupancy grids, but advancements in lidar sensors, occupancy prediction, and BEV detectors are helping improve autonomous vehicle perception and safety. As autonomous driving systems evolve, 3D detection continues to be crucial for precise navigation and decision-making.
Topic 6 - Weakly Supervised Object Segmentation: Balancing Annotations and Performance (244 documents) Weakly supervised object segmentation focuses on leveraging less detailed annotations, such as image-level labels or object proposals, to train segmentation models. Unlike fully supervised methods that require pixel-level annotations, weak supervision relies on class-level or bounding box labels to guide the segmentation process. Datasets like Pascal VOC and MS COCO provide benchmarks for evaluating segmentation models, with metrics such as mean Intersection over Union (mIoU) used to assess performance. Techniques like class-agnostic object masks, discriminative region mapping (CAM), and pseudo-masks are employed to generate pixel-level segmentations from weak annotations. This approach aims to reduce the cost and effort associated with obtaining high-quality, pixel-wise annotations, while still achieving competitive segmentation results, especially for complex tasks like instance-level segmentation and object part identification.
Topic 7 - Image Relighting: Enhancing Lighting and Material Effects in Digital Rendering (167 documents) Image relighting is a technique used in computer graphics and computational photography to manipulate or simulate changes in lighting conditions on a given scene or object. This process involves adjusting various aspects of lighting, such as specular and diffuse reflections, albedo, and shadow effects, to create realistic or desired lighting outcomes. It takes into account material properties like reflectance, illumination, and surface normals, which determine how light interacts with the scene. Relighting is commonly applied in tasks such as portrait or face relighting, where the goal is to alter lighting without changing the geometry of the scene. By estimating and adjusting lighting effects like specular highlights, shadows, and ambient light, image relighting allows for enhanced visual realism and flexibility in various applications, from film production to virtual environments and interactive systems.
Topic 8 - Large Kernel Convolutions and Self-Attention Mechanisms in Vision Transformers (209 documents) The integration of large kernel convolutions and self-attention mechanisms has become a powerful approach in modern computer vision tasks. Large kernel convolutions, such as atrous or depthwise convolutions, allow for an increased receptive field, enabling the model to capture long-range dependencies in an image without significantly increasing computational cost. This is essential for vision tasks like object detection and segmentation, where understanding the global context is crucial. On the other hand, self-attention mechanisms, particularly in Vision Transformers (ViTs), facilitate capturing relationships between distant image patches, enhancing the model's ability to focus on relevant parts of an image. By combining large kernel convolutions with self-attention layers, models like the Vision Transformer (ViT) can effectively balance local feature extraction and global context understanding, leading to improved performance on benchmarks like ADE, Cityscapes, and COCO, especially in tasks like segmentation and scene understanding. This combination provides an efficient and scalable solution for handling complex vision tasks while maintaining competitive performance.
Topic 9 - 3D-Aware Image Synthesis with GANs for High-Fidelity and Controllable Rendering (71 documents) 3D-aware image synthesis is an advanced technique that combines the power of Generative Adversarial Networks (GANs) with 3D geometry to create highly realistic and controllable images. By incorporating 3D-aware models like Neural Radiance Fields (NeRF) and leveraging latent spaces in GAN architectures such as StyleGAN, this method enables the generation of high-fidelity images from novel views or multi-view perspectives. This approach allows for fine-grained control over attributes like lighting, angles, and details, making it particularly useful for photorealistic rendering and editing. The synthesis process ensures that the generated images maintain consistency across different views and provide high-quality visual outputs, which can be applied in areas such as virtual reality, digital content creation, and computer graphics. With advancements in 3D-aware GANs, it is now possible to synthesize photo-realistic images with impressive fidelity, enabling novel applications in creative industries.
Topic 10 - Efficient Solving of Non-Convex Problems with Outlier Rejection and Relaxation Techniques (356 documents) Solving non-convex problems, especially those involving outlier detection, correspondence, and registration, is a complex challenge in fields like computer vision and robotics. Methods like RANSAC (Random Sample Consensus) and polynomial solvers are commonly used to handle these issues, where outliers — incorrect data points — are filtered out to improve the accuracy of the solution. Convex relaxation techniques and non-convex optimization solvers are applied to iteratively refine the solution toward a globally optimal result. For example, pose estimation problems, such as relative rotation or translation, are solved efficiently by leveraging convex optimization and minimal solvers, ensuring that even with noisy or incomplete data, the solution converges to the correct answer. These techniques, including the use of least squares and graph matching, play a key role in ensuring robust performance in real-world applications, where noise and outliers are unavoidable.
In this analysis, we have explored the trends and shifts in research topics within the CVPR community over the past years. The data reveals a dynamic landscape, with certain areas experiencing significant growth while others have seen a decline in interest. This reflects the evolving nature of artificial intelligence research and the continuous pursuit of innovation and improvement in various domains.
The hottest topics — ranging from instruction-tuned multimodal LLMs for vision-language understanding to the rapid advancements in text-guided editing, 3D-aware synthesis, and event-based vision — highlight a strong drive toward bridging modalities, creating more controllable generative models, and addressing the growing needs of real-world applications. Researchers are increasingly focusing on improving the integration of language and vision to enable more effective reasoning, better handling of ambiguities (such as hallucinations), and enhanced performance in both creative and safety-critical environments.
In contrast, the coldest topics — such as self-supervised pretraining, traditional vision-language alignment, semi-supervised learning, domain adaptation, and even classical 3D object detection — indicate areas where mature techniques have plateaued. While these methods laid the foundations for current advances, their rate of improvement appears to have slowed in favor of newer approaches. Techniques once at the cutting edge are now being revisited with an eye toward integrating them into more comprehensive systems, but their standalone appeal is declining as the community shifts toward end-to-end, multimodal, and task-specific solutions.
Taken together, the trends suggest that the field is steering toward more holistic, integrated models that not only push the boundaries of what automated systems can generate or analyze but also provide greater reliability and control in real-world applications. As the industry continues to explore the fusion of text, image, and even sensor data, the next wave of innovation will likely be driven by systems that learn from multiple modalities concurrently — while leveraging long-standing, robust principles as a stepping stone.
This evolution underscores the vibrant nature of artificial intelligence research, where established methods provide a stable base while emerging techniques hold the promise of reshaping the future of artificial intelligence.
]]>sli.dev for non-web developers2023-06-01T17:19:10+00:002023-06-01T17:19:10+00:00https://george-gca.github.io/blog/2023/slidev_for_non_web_devsI always struggled every time I had to do a new presentation. Don’t get me wrong, Google Slides is good, and suffices for most common use cases. The problem is when the slides keeps changing, and you want to do some versioning on them. I made some slides in the past using LaTeX with the Beamer class. A friend even showed me a nicer template than the default ones. The problem is, though this solution is portable (you can generate a PDF file), it lacks some features that I wanted, like animations, transitions, and support for drawing.
I took a look at Reveal.js, but it was too much work to setup and maintain. I also went for Remark, but the last update was more than 2 years ago. That’s when I found sli.dev. It’s a framework under constant development for creating slides using Markdown. It’s based on a lot of web technologies, but you don’t need to be fluent on them to use it. Here is everything that you’ll need to get started.
Installing node version manager (nvm)
nvm allows you to quickly install and use different versions of node via the command line. To install the latest version, run the following command:
get_latest_github_raw_no_v(){# use it for when the link for the download doesn't have a version included in the name of the file# e.g.: https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.3/install.sh# usage: get_latest_github_raw_no_v "user/repo" "filename"version=$(curl --silent"https://api.github.com/repos/$1/releases/latest" | # Get latest release from GitHub apigrep'"tag_name":' | # Get tag linesed-E's/.*"([^"]+)".*/\1/')# Pluck JSON valueecho"https://raw.githubusercontent.com/$1/$version/"$2}# Install node version manager (nvm)site=$(get_latest_github_raw_no_v "nvm-sh/nvm""install.sh")
curl -o-$site | bash
Then, add the following lines to your ~/.bashrc file:
# enable node version managerif[-d"$HOME/.nvm"];then
export NVM_DIR="$HOME/.nvm"[-s"$NVM_DIR/nvm.sh"]&&\."$NVM_DIR/nvm.sh"# This loads nvm[-s"$NVM_DIR/bash_completion"]&&\."$NVM_DIR/bash_completion"# This loads nvm bash_completionfi
Restart your terminal, or just reload the ~/.bashrc file:
. ~/.bashrc
Installing Node (latest version)
To install the latest node version, run the following command:
In my case, the latest version available is v20.2.0. You can check the installed version by running:
node --version
Installing sli.dev
To install sli.dev, simply run:
npm init slidev
This command will install everything you need, ask for the project name, and start the template project. After the installation, you should see the following lines in your terminal:
public slide show > http://localhost:3030/
presenter mode > http://localhost:3030/presenter/
remote control > pass --remote to enable
shortcuts > restart | open | edit
If the browser did not open automatically, you can open it manually by accessing the URL http://localhost:3030/. To stop running the server, just press Ctrl+C in the terminal. After stopped, you can start the development server again by entering the created directory and running:
npm run-script dev
To see a list of scripts, run npm run:
Scripts available in XXXXXXXXX via `npm run-script`:
build
slidev build
dev
slidev --open
export
slidev export
If you want to modify any of these commands (or create more, like exporting presenter notes), you can edit the package.json file. For example, to add a command to export presenter notes, add the following lines to the scripts section:
"export_notes":"slidev export-notes"
Changing sli.dev template
You can change the presentation theme simply by editing the theme attribute on the front matter of slides.md file. When you change and save it, the cli interface will automatically download and apply the new theme.
theme:academic
sli.dev installing new theme.
Basic settings
I am not a hardcore sli.dev user or web developer. So there were some things that took some time for me to figure out, and I think it is worth mentioning. The first one is forcing the slides to be in dark mode. To do that, add the following line to the front matter of slides.md:
colorSchema:"dark"
While using the academic theme, by default every slide with a # title will be added to the table of contents. To hide a slide from the table of contents, add the following line to the front matter of the slide:
hideInToc:true
For the sake of organization, I like to create a different file for each section in my presentation, then import these files in my main slides.md. To do that, create a new file with the markdown extension, then add the following lines to slides.md for each file to be included:
---src:slides/other_slide.md---
Exporting slides
Exporting to PDF or PNG relies on Playwright for rendering. You will therefore need to install playwright-chromium to use this feature. For this, run:
npm i -D playwright-chromium
Then, add the following lines to the scripts section of the package.json file:
To check for dependencies that need to be updated, run:
npm outdated
which will output something like this:
Package Current Wanted Latest Location Depended by
@slidev/cli 0.43.11 0.43.15 0.46.1 node_modules/@slidev/cli xxxxxxxx
To updated installed dependencies, simply run:
npm update
Keep in mind that npm update will never update to a major breaking-changes version, only to a minor one. What this means is, it will use the Wanted version in the table above instead of the Latest version. To obtain the Latest version, call the install command with @latest appended to the package name. For example, to update @slidev/cli to the latest version, run:
npm install @slidev/cli@latest
]]>Improving your python code with simple tricks2023-01-25T13:28:15+00:002023-01-25T13:28:15+00:00https://george-gca.github.io/blog/2023/simple-improvements-python-codeSometimes our code is running slow. Sometimes it is eating up memory. Maybe it is just not as readable as we would like it to be. In this post, we will see how to use some functions from the default library to improve our code. All the code used in this post is available here. While I only presented a few functions that I use frequently, there are many more that can be used to improve your code. I encourage you to check the official documentation to see what else is available.
The list comprehension executes the loop in Python bytecode, just like a regular for loop. The list() call iterates entirely in C code, which is far faster.
To compare all these solutions, lets check the equivalent bytecodes. For the loop solution:
To create a generator like a list comprehension (called generator expression), just replace the squared brackets [ ] with parenthesis ( ).
Why does it matter?
Generators looks like a normal function, except that it contains yield expressions for producing a series of values usable in a for-loop or that can be retrieved one at a time with the next() function. It returns a generator iterator, which temporarily suspends processing, remembering the location execution state (including local variables and pending try-statements).
Let’s do some comparisons:
tmp=sum([iforiinrange(10_000_000)])
860 ms ± 30.4 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
tmp=sum((iforiinrange(10_000_000)))
609 ms ± 2.93 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
Not that different right? Now, let’s check the memory usage. Let’s focus on increment, since it represents the difference in memory between the beginning and end of this execution.
memory increment: 263.44 MiB
memory increment: 0.00 MiB
What happened? The generator only returns one element at a time, which is given to the sum function. This way, we don’t need to pre-generate the whole list to perform the sum of the elements. In fact, since the sum function gets an iterator as parameter, we could call it like this:
tmp=sum(iforiinrange(10_000_000))
593 ms ± 90.4 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
memory increment: 0.01 MiB
Or even:
tmp=sum(range(10_000_000))
168 ms ± 4.68 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
memory increment: 0.00 MiB
Which runs way faster.
Avoid generating all the values whenever possible
For the sake of these examples, suppose we have an ordered list of values.
What to do if we want only the values lower than a limit?
The list comprehension way of achieving this is this:
tmp=[iforiinrange(10_000_000)ifi<1_000_000]
596 ms ± 7.16 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
memory increment: 0.02 MiB
116 ms ± 2.37 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
memory increment: 0.01 MiB
Why is it faster with loops?
Because using list comprehension, the whole list must be generated before selecting the elements. The same is not true for the loop, that only runs through some of the values.
1.06 s ± 8.41 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
memory increment: 0.00 MiB
In this case it is quicker to generate the whole list, and then filter it. But note, while using list comprehension is quicker, takewhile takes less memory, since it still doesn’t need to store the whole list, even momentarily.
What if we want only the values higher than a limit?
1.3 s ± 93.3 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
memory increment: 0.15 MiB
Now, with list comprehension:
tmp=[iforiinrange(10_000_000)ifi>1_000_000]
978 ms ± 11.2 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
memory increment: 169.84 MiB
In this case, since the loop will run through every element, it is slower than the list comprehension. It takes less memory though, since it doesn’t need to store the whole list in memory.
147 ms ± 3.09 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
memory increment: 0.00 MiB
Doing it with list comprehension:
tmp=[iforiinrange(10_000_000)][:1_000_000]
523 ms ± 10.8 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
memory increment: 0.01 MiB
Why is it faster with loops?
Because using list comprehension, the whole list must be generated before doing the slice operation. The same is not true for the loop, that only runs through some of the values.
796 ms ± 11.8 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
memory increment: -0.16 MiB
Note again that, as it happened with dropwhile when the condition takes longer to be satisfied, while using islice is slower than doing it with list comprehension, it takes much less memory.
What if we just want to count the number of elements that will be generated?
Suppose we want to know how many elements will be generated from a condition. Usually, we would do it like this:
tmp=[iforiinrange(value)ifi%2==0]count=len(tmp)
1.02 s ± 63.6 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
increment: 0.04 MiB
But the problem is, we are storing a whole list in memory only to get its length.
Can we do better?
Yes, by creating a generator that generates 1 every time the condition is true, and summing it.
count=sum(1foriinrange(value)ifi%2==0)
991 ms ± 18.9 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
increment: 0.00 MiB
Let’s take the fibonacci function, for example, that calls itself recursively.
deffib(n):ifn<2:returnnreturnfib(n-1)+fib(n-2)
Let’s use it in a list comprehension to get the first 16 fibonacci numbers:
tmp=[fib(i)foriinrange(16)]
698 µs ± 152 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
It takes a pretty good amount of time to execute for a small number. But what if we could automatically save the results of the previous calls to the function? That is what lru_cache is all about. It stores previous calls, with its given parameters and calculated output, as a least recent used (LRU) cache. This way, whenever we call the function and this call was already made (and its results are still stored in the cache), we simply get the results from the cache.
Suppose we have a function, called divide_by. It is a pretty generic function, but it is usually called with some specific values, like dividing by two, or by three.
What if, instead of creating an entire new function, we could only create different signatures for the function, one for each common y value? That is what partial is for:
]]>The problem of research code reproducibility2022-10-21T15:13:16+00:002022-10-21T15:13:16+00:00https://george-gca.github.io/blog/2022/research-code-reproducibilityEveryone in the research community had that problem already:
You found a paper that you want to reproduce. You found out that the original author (or possibly another researcher) published code to run those experiments. You download the code, try to run it, but it doesn’t work. There are missing code, missing datasets, or missing dependencies. Maybe the code is a little outdated, and you need to update it to run it on your machine. Or you need to find out which specific library versions the authors used. You spend hours trying to get the code to run, but to no avail. You try to contact the authors, but they do not respond. You try to fix the code yourself, but it is too complicated. You give up.
Who never got so frustrated to the point that you just give up on that code, maybe even on that paper? Hell, I know people that gave up on a whole field of research because they could not reproduce the results of a paper. Multiple times I got caught in this kind of situation, and got me questioning the whole code publishing part. After a lot of afterthought, I came to a partial solution to this problem.
The main culprit
Most of the time that I am unable to run someone else’s code, it is because of one thing: dependencies versions are not specified.
Sure, a lot of people create something like requirements.txt, but they usually lack the version number. Even if they do specify the version number, it is usually not the exact version number. For example, instead of numpy==1.19.5, they would write numpy>=1.19.5. This is not good enough. If you want to reproduce someone else’s results, you need to know exactly which version of each dependency you need to install. Otherwise, you will get different results. Even if you specify the exact version number, you might still get different results if the authors used a different version of the language, say, Python, or some other low-level dependencies, like cuda or cuDNN.
Docker for the help
That is where Docker comes in. Docker is a containerization technology that allows you to create an image that packs all the dependencies that you need to run your code. You can then run your code inside a container, and you will possibly get the same results as the authors. Making an allusion to object oriented programming, image is the class and container is an object or an instance from that class. You can even share that image with other people, so they can run your code too. Docker also supports running code on GPUs.
As an example, here is the base code that I used during my master degree. I tried as much as possible to make it reproducible and extensible. That means to not only run the AI model, but also be able to create your own model and run it with the same data and training parameters, to allow comparisons.
I decided to split the logic of dealing with Docker (contained in Makefile) from running the Python code itself (contained in start_here.sh). The recipe for creating the Docker image is inside the Dockerfile, and I also made available a Dockerfile with all the versions of the dependencies fixed, like you can see in the code snippet below. Everything you might need to run the code is explained in the README file.
To ensure reproducibility despite the random number generators, I used the seed_everything function from PyTorch Lightning. This function sets seed for pseudo-random number generators in: pytorch, numpy, python.random. It also sets the following environment variables: PL_GLOBAL_SEED and PL_SEED_WORKERS.
Docker doesn’t solve everything
As you can probably be thinking, this is not a perfect solution. There are still problems that might surge, and that you need to be aware of, since some of these are from out of our control.
# temporary solution for bug# see https://forums.developer.nvidia.com/t/gpg-error-http-developer-download-nvidia-com-compute-cuda-repos-ubuntu1804-x86-64/212904/3
apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64/3bf863cc.pub &&\
apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/3bf863cc.pub &&\
This is a solution for a bug that I found with the NVIDIA repository. Basically, they are (were?) in the process of rotating their repository GPG public keys, which is used to sign the packages. This was causing an error in the apt-get update command when creating the Docker image, and this was not happening during the entirety of my experiments, but happened by the time I decided to publish this code.
Dependencies’ dependencies
Sometimes some of your dependencies specific version requires specific versions of its own dependencies. This happened to me when specifying tensorboard==2.9.0, which caused an error when building the Docker image. To fix this, I needed to define the proper protobuf version to install, even though my code does not use protobuf directly.
prettytable==3.3.0 \# specify protobuf to avoid bug with tensorboard# https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updatesprotobuf==3.20 \
pytorch-lightning==1.6.3 \tensorboard==2.9.0 \
]]>Creating localized blog posts2022-09-30T17:40:13+00:002022-09-30T17:40:13+00:00https://george-gca.github.io/blog/2022/localized-blogThis post is part of a series of posts that explain how to set up your own site based on the al-folio theme and add support for a second language:
If you go to the blog section of your al-folio site, you’ll realize that it is quite empty, although there are posts in the template. Actually, the Jekyll Multiple Languages Plugin already support localized blog posts. It is not displaying them because it is not finding them. So, let’s create the correct structure for them. Create a _posts/ directory inside each language under the _i18n/ directory, and copy the content of the _posts/ directory from the root of the website to the language directories. So, for example, if you have a _posts/ directory with the following content:
_posts/2015-03-15-formatting-and-links.md
_posts/2015-05-15-images.md
_posts/2015-07-15-code.md
_posts/2015-10-20-comments.md
_posts/2015-10-20-math.md
_posts/2018-12-22-distill.md
_posts/2020-09-28-github-metadata.md
_posts/2020-09-28-twitter.md
_posts/2021-07-04-diagrams.md
_posts/2022-02-01-redirect.md
You should create the following structure for all your languages, in this example, the English language:
Create that, translate the pages contents, and it will now show the missing blog posts. Easy, right? Errr, you are missing a few small details: the date format, the reading time, and the link when clicking on a tag in the header of the blog section. The bad news is that Jekyll doesn’t natively support localized date formats. The good news is that it is not that hard to create it though. Let’s start with the date format.
Localizing the date format
There are two main formats of date used in the template: in the posts list and inside the posts, with the format September 28, 2020, and when filtering the posts (e.g.: by tag), with the format Sep 28, 2020. The easiest way I found of localizing them is by first manually translating the months names. For this, add the following section to your language files (_i18n/en.yml and _i18n/pt-br.yml), translating the months names to your language:
Since the dates are used in a variety of locations, let’s create a function to reuse the code. Create the file _includes/date_format.html with the following code:
{%assignmonths='january|february|march|april|may|june|july|august|september|october|november|december'|split:'|'%}{%assignm=include.date_from.date|date:'%-m'|minus:1%}{%assignday=include.date_from.date|date:'%d'%}{%capturemonth%}months.{{include.format}}.{{months[m]}}{%endcapture%}{%assignyear=include.date_from.date|date:'%Y'%}{%ifsite.lang=='en'-%}{%-tmonth%}{{day}}, {{year-}}{%-else-%}{{-day}} de {%tmonth%}, {{year-}}{%-endif%}
This code defines a function that extracts the month, day, and year from the date_from variable, formats according to format, and returns the formatted localized string. Notice that, to access the given variables, we must refer to them preceded with include.. The format variable can be either long or short, as we defined above, and the date_from variable must have a date object inside it. The function also considers the current language. Now, let’s call the function with the proper parameters. Inside the file blog/index.html, do the following changes:
The posts list will now display the date format correctly. Just do the same changes for every other place where the date format appears, like the filters pages, changing the month format to "short" when needed, and also giving the correct date_from parameter. For example, for the file _layouts/archive-category.html the change will be:
Now, let’s localize the reading time. Do the following changes to the file blog/index.html:
<!-- {{ read_time }} min read · -->
{% if site.lang == 'en' %}{{ read_time }} min read{% else %}Leitura de {{ read_time }} min{% endif %} ·
Fixing blog archive navigation
When you click to filter blog posts by tag, year, or category, the page will show the posts, but the navigation will be broken. This is because these navigations are not localized. To fix this, modify the following lines in files _layouts/archive-category.html, _layouts/archive-tag.html, and _layouts/archive-year.html:
Also, change the following line in the file blog/index.html:
<!-- <i class="fas fa-hashtag fa-sm"></i> <a href="{{ tag | prepend: '/blog/tag/' | relative_url }}">{{ tag }}</a> --><iclass="fas fa-hashtag fa-sm"></i><ahref="{{ tag | prepend: '/blog/tag/' | prepend: site.baseurl }}">{{ tag }}</a>
Fixing pagination
If your blog has enough posts to enable more pages with results (pagination), you’ll realize that it is not yet translated. To fix this, first we need to create localized words for Older and Newer in the language files _i18n/en.yml and _i18n/pt-br.yml, respectivelly:
pagination:newer:Newerolder:Older
pagination:newer:Recentesolder:Antigas
Next, we change all uses of relative_url for prepend: site.baseurl in the file _includes/pagination.html, so it correctly handles the language urls. Also, change the words “Newer” and “Older” for its correspondent translation from the correct language file:
If you filter your blog posts by year, you’ll notice that the year is not displayed in the browser title. To fix this, modify the following lines in the file _includes/metadata.html:
]]>Creating localized Projects pages2022-09-30T11:57:13+00:002022-09-30T11:57:13+00:00https://george-gca.github.io/blog/2022/localized-projectsThis post is part of a series of posts that explain how to set up your own site based on the al-folio theme and add support for a second language:
Let’s first localize all the pages descriptions. This will start with us creating the keys in the _i18n/en.yml and _i18n/pt-br.yml files, and moving there the referred content from the pages. So, let’s start by creating the descriptions both in the file root and inside the projects key. The new _i18n/en.yml content will be:
titles:about:aboutblog:blogcv:cvnews:newsprojects:projectspublications:publicationsrepositories:repositoriesteaching:teachingsubmenus:submenusunk:page not founddescriptions:about:about meblog:bloggingpublications:publications by categories in reversed chronological order. generated by jekyll-scholar.projects:A growing collection of your cool projects.repositories:Edit the `_data/repositories.yml` and change the `github_users` and `github_repos` lists to include your own GitHub profile and repositories.teaching:Materials for courses you taught. Replace this text with your description.unk:Looks like there has been a mistake. Nothing exists here.projects:titles:project1:Project 1project2:Project 2project3:Project 3project4:Project 4project5:Project 5project6:Project 6descriptions:project1:a project with a background imageproject2:a project with a background imageproject3:a project that redirects to another websiteproject4:another without an imageproject5:a project with a background imageproject6:a project with no image
And now for the _i18n/pt-br.yml:
titles:about:sobreblog:blogcv:cvnews:novidadesprojects:projetospublications:publicaçõesrepositories:repositóriosteaching:ensinosubmenus:submenusunk:página não encontradadescriptions:about:sobreblog:bloggingpublications:publicações por categoria em ordem cronológica reversa. gerado pelo jekyll-scholar.projects:Uma crescente coleção de seus projetos interessantes.repositories:Edite o `_data/repositories.yml` e mude as listas `github_users` e `github_repos` para incluir seu próprio perfil do GitHub e repositórios.teaching:Materiais de cursos que você ministrou. Substitua esse texto com sua descrição.unk:Parece que ocorreu um erro. Não existe nada aqui.projects:titles:project1:Projeto 1project2:Projeto 2project3:Projeto 3project4:Projeto 4project5:Projeto 5project6:Projeto 6descriptions:project1:um projeto com imagem de fundoproject2:um projeto com imagem de fundoproject3:um projeto que redireciona pra outro websiteproject4:outro sem imagemproject5:um projeto com imagem de fundoproject6:um projeto sem imagem
Next, update all the pages to use the new keys. Those will be: 404.html, _pages/projects.md, _pages/publications.md, _pages/repositories.md, _pages/teaching.md, and all projects inside _projects/ directory. For example, the new description in 404.html will be:
description: descriptions.unk
and in _projects/1_project.md:
description: projects.descriptions.project1
Also, update _layouts/cv.html and _layouts/page.html to only display the translation description if a description is defined in the markdown file. The new _layouts/page.html will have:
<!-- <p class="post-description">{{ page.description }}</p> --><pclass="post-description">{% if page.description %}{% t page.description %}{% endif %}</p>
Now the descriptions should also be localized. The titles and the descriptions of the projects when a project is opened should also be localized, but not in the projects overview page. So, let’s fix this.
Projects overview still not working.
Localizing projects overview
When you look inside _pages/projects.md, you will see more code than in your regular markdown file. The authors decided to call here the html code for building the main layout of the overview page. The interesting parts are these ones:
{% if page.horizontal -%}
<divclass="container"><divclass="row row-cols-2">{%- for project in sorted_projects -%} {% include projects_horizontal.html %} {%- endfor %}</div></div>
{%- else -%}
<divclass="grid">{%- for project in sorted_projects -%} {% include projects.html %} {%- endfor %}</div>
{%- endif -%}
These parts mean that there is a horizontal and a vertical layout for the projects overview page. The horizontal layout is used when the horizontal key is set to true in the header of this page, and the vertical otherwise. These layouts can both be found in _includes/projects_horizontal.html and _includes/projects.html, respectively. These are the files that we need to update to localize the projects overview page. The changes will basically be the same on both files: we need to add the t tag to the titles and descriptions of the projects, and also correctly update the links to them. The changes to the _includes/projects_horizontal.html will be:
Now, the projects overview is displayed correctly. But, if you look closely, you’ll notice that the projects categories have not been translated. Let’s fix this.
Projects overview now working.
Localizing projects categories
The categories will only be translated when displaying, not inside the projects’ headers. This means that, when creating a project, you will still use the categories in English, like in _projects/1_project.md:
category:work
For this, create the localized version of the categories for both languages. Let’s keep this inside the projects key, so that it will now look like this in _i18n/en.yml:
projects:titles:project1:Project 1project2:Project 2project3:Project 3project4:Project 4project5:Project 5project6:Project 6descriptions:project1:a project with a background imageproject2:a project with a background imageproject3:a project that redirects to another websiteproject4:another without an imageproject5:a project with a background imageproject6:a project with no imagecategories:fun:funwork:work
Now, in _pages/projects.md, we need to get the correct category inside the loop, then its translation. For this, we’ll use the capture tag. The new code will be:
Now, everything is localized. The projects overview page, the projects categories, and the projects pages.
Projects overview with localized categories.
]]>Creating localized CV pages2022-09-29T21:40:13+00:002022-09-29T21:40:13+00:00https://george-gca.github.io/blog/2022/localized-cvThis post is part of a series of posts that explain how to set up your own site based on the al-folio theme and add support for a second language:
The current structure of the cv page is composed of 3 main files: _pages/cv.md, _layouts/cv.html, and _data/cv.yml. The first one is the definition of the page, the second one is the layout of the page, and the third one is the data that is used to populate it. We will create a new folder for each language inside the _data/ directory, and copy the file _data/cv.yml to both of them. The new _data/ directory will look like this:
_data/en/cv.yml
_data/pt-br/cv.yml
_data/coauthors.yml
_data/repositories.yml
_data/venues.yml
Now, lets replace the content of the _data/pt-br/cv.yml file with the following:
-title:Informações Geraistype:mapcontents:-name:Nome Completovalue:Albert Einstein-name:Data de Nascimentovalue:14 de março de 1879-name:Idiomasvalue:Inglês, Alemão-title:Educaçãotype:time_tablecontents:-title:PhDinstitution:University of Zurich, Zurique, Suíçayear:1905description:-Descrição 1.-Descrição 2.-title:Descrição 3.contents:-Sub-descrição 1.-Sub-descrição 2.-title:Diploma de ensino federalinstitution:Eidgenössische Technische Hochschule, Zurique, Suíçayear:1900description:-Descrição 1.-Descrição 2.-title:Experiênciatype:time_tablecontents:-title:Professor de Física Teóricainstitution:Institute for Advanced Study, Princeton Universityyear:1933 - 1955description:-Descrição 1.-Descrição 2.-title:Descrição 3.contents:-Sub-descrição 1.-Sub-descrição 2.-title:Visiting Professorinstitution:California Institute of Technology, Pasadena, Califórnia, EUAyear:1933description:-Descrição 1.-Descrição 2.-title:Diretorinstitution:Kaiser Wilhelm Institute for Physics, Berlim, Alemanha.year:1917-1933-title:Professor de Física Teóricainstitution:Karl-Ferdinand University, Praga, Tchecoslováquiayear:1911 - 1917description:-title:Professor Associado de Física Teóricainstitution:University of Zurich, Zurique, Suíçayear:1909 - 1911-title:Projetos de Código Abertotype:time_tablecontents:-title:<a href="https://github.com/alshedivat/al-folio">al-folio</a>year:2015-atualdescription:Um tema Jekyll bonito, simples, limpo e responsivo para acadêmicos.-title:Honras e Prêmiostype:time_tablecontents:-year:1921items:-Prêmio Nobel de Física-Medalha Matteucci-year:2029items:-Medalha Max Planck-title:Interesses Acadêmicostype:nested_listcontents:-title:Tópico 1.items:-Descrição 1.-Descrição 2.-title:Tópico 2.items:-Descrição 1.-Descrição 2.-title:Outros Interessestype:listcontents:-<u>Hobbies:</u> Hobby 1, Hobby 2, etc.
Now, open the _layouts/cv.html file and replace the following line:
<!-- {% for entry in site.data.cv %} -->
{% for entry in site.data[site.lang].cv %}
This will be sufficient to make the page work for the new language. Now, lets add the option to download the localized pdf file. For this, create a new folder for each language inside the assets/pdf/ directory, and copy the file assets/pdf/example_pdf.pdf to both of them. The new assets/pdf/ directory will look like this:
assets/pdf/en/example_pdf.pdf
assets/pdf/pt-br/example_pdf.pdf
assets/pdf/example_pdf.pdf
Now, lets update the link to the localized pdf file. We will use the same name for both files, which is already defined in the cv_pdf: example_pdf.pdf attribute inside _pages/cv.md. If you wish to use a different name for the pdfs, rename both and also update this information in the related attribute. Next, modify the following line in the _layouts/cv.html file:
]]>Turning your al-folio into a dual-language website2022-09-28T11:29:13+00:002022-09-28T11:29:13+00:00https://george-gca.github.io/blog/2022/dual-language-al-folioThis post is part of a series of posts that explain how to set up your own site based on the al-folio theme and add support for a second language:
al-folio is a great theme for Jekyll websites. It is very customizable and easy to use, with support for blogging, repository information, projects, and more (see demo). However, it does not support multiple languages out of the box. This post will show you how to add support for another language in your al-folio. I’ll add support for pt-BR, since it is my mother language. I’ll assume you have already cloned your copy of the al-folio repository and have it running locally.
Installing dependencies
We will do this with the help of the Jekyll Multiple Languages Plugin. It adds i18n support for Jekyll. To install it, add the following line to your Gemfile under the group :jekyll_plugins do:
gem'jekyll-multiple-languages-plugin'
Also, add the following line to your _config.yml under plugins::
-jekyll-multiple-languages-plugin
and the following lines after it (outside plugins), for example, before the Jekyll Minifier section:
# multi language settingslanguages:["en","pt-br"]default_locale_in_subfolder:false
Setting default_locale_in_subfolder to false will make your main language be the root of your website, instead of being in a subfolder. For example, instead of https://george-gca.github.io/en/, it will be https://george-gca.github.io/. This is the default behavior of al-folio, so we will keep it. The first language in the list will be the default language, English in this case. Then, run bundle install to install the plugin.
Creating translation files
Create a folder called _i18n and add sub-folders for each language, using the same names used on the languages setting on the _config.yml we just added. Also, create a yml file for each language. For example, for pt-br, create a folder called _i18n/pt-br. Then, create a file called _i18n/pt-br.yml. Our directory structure should look like this:
_i18n/en.yml
_i18n/pt-br.yml
_i18n/en/
_i18n/pt-br/
Adding language toggle
Now, we need to add a language toggle to our website. We will add it to the navigation bar. Open the file _includes/header.html and add the following code before the Toogle theme mode area:
<!-- Toogle language --><liclass="nav-item active">
{% if site.lang == "en" %}
<aclass="nav-link"href="{{site.baseurl_root}}/pt-br{{page.url}}"> PT-BR </a>
{% elsif site.lang == "pt-br" %}
<aclass="nav-link"href="{{site.baseurl_root}}{{page.url}}"> EN </a>
{% endif %}
</li>
This will add a link to the other language. The page.url will keep the current page, so the user will not be redirected to the home page. Note that site.baseurl_root is a variable introduced by the Jekyll Multiple Languages Plugin, and it points to the root of the page without the language path. More information about the newly added variables can be found here.
Header with language toggle.
Adding translated titles
Until now, we added everything we needed to support the translation, but haven’t done the translation per se. We will start that now. Open the file _i18n/en.yml and add the following lines:
titles:about:aboutblog:blogcv:cvnews:newsprojects:projectspublications:publicationsrepositories:repositoriesteaching:teachingsubmenus:submenusunk:page not found
This will add the titles for the pages. Now, open the file _i18n/pt-br.yml and add the following lines:
titles:about:sobreblog:blogcv:cvnews:novidadesprojects:projetospublications:publicaçõesrepositories:repositóriosteaching:ensinosubmenus:submenusunk:página não encontrada
This will add the titles for the pages in Portuguese. Now that we have the translated titles, we have to tell the pages to use these instead of the hardcoded ones. To do so, open all the pages under the _pages folder and change their title to use the correct titles variable. For example, the new title for the about.md page should look like this:
title:titles.about
If you run your website now, you’ll see that the titles are shown as titles.about instead of just about as it was supposed to, since its default is in English. We still need to tell the html templates to select the correct translated version of these variables. To do so, open the file _includes/header.html and change all the title variables to use the t function. The t function, or its longer version translate, will ensure that it will select the correct version from the current language yml file. More specifically, do the following changes:
Now run your website again and you’ll see that the titles have the correct values. You can even change the language on the toggle and see that the titles change accordingly.
Header in Portuguese.
Fixing translated navigation
Everything seems to work fine, except not. If you click in the PT-BR toggle in the header, then click in another page (e.g. repositories), it will change again the titles to English. To handle this kind of situation, we have to make sure that when the user clicks the header links, it will keep the language. Lets do this first for the about page. To do so, open the file _includes/header.html and change the link to the about page to use the site.baseurl variable instead of the relative_url. More specifically, do the following changes:
<!-- <a class="nav-link" href="{{ '/' | relative_url }}">{% t about_title %} --><aclass="nav-link"href="{{ '/' | prepend: site.baseurl}}">{% t about_title %}</a>
Now, if you are in another page, for example, the repositories page, and click in the PT-BR toggle, then click in the about page, it will keep the language in the titles. Now, let’s enable this for all the other pages. To do so, open the file _includes/header.html again and do the following changes:
Now everything is working fine! The sun is rising, and the world is colorful again :rainbow:. But wait, there is still stuff missing. If you click in the PT-BR toggle, then click in the publicações page, you’ll see that the title of the page is still titles.publications.
What should be a translated title.
Oh crap, will this ever end someday? :scream: Well, yes, it will. We just need to tell the pages to use the correct translated title. To do so, let’s see which templates do the pages use. The page publications for example, which is located in file _pages/publications.md, uses the template page, as can be seen by the following line in the beginning of the file:
layout:page
So, lets open the file _layouts/page.html and change the title to use the t function. More specifically, do the following changes:
<!-- <h1 class="post-title">{{ page.title }}</h1> --><h1class="post-title">{% t page.title %}</h1>
Now, if you run your website, it will not work. This happens because the t function is now trying to translate a variable that is not defined. But where? If you do a little search, you’ll notice that the publications page is not the only one that uses the layout: page. All pages that use it are:
404.html
news.html
_pages/dropdown.md
_pages/projects.md
_pages/publications.md
_pages/repositories.md
_pages/teaching.md
_projects/1_project.md
_projects/2_project.md
_projects/3_project.md
_projects/4_project.md
_projects/5_project.md
_projects/6_project.md
You need to change the title for ALL these pages. We already did it for the pages inside _pages/, so let’s do it for the rest. Open the file 404.html and change its title: "Page not found" to title: titles.unk, since we already defined titles.unk in both _i18n/en.yml and _i18n/pt-br.yml. To keep the projects section more organized, let’s create new attributes for it inside each of the translation files. So, the new _i18n/en.yml and _i18n/pt-br.yml will look like this, respectively:
titles:about:aboutblog:blogcv:cvnews:newsprojects:projectspublications:publicationsrepositories:repositoriesteaching:teachingsubmenus:submenusunk:page not foundprojects:titles:project1:Project 1project2:Project 2project3:Project 3project4:Project 4project5:Project 5project6:Project 6
titles:about:sobreblog:blogcv:cvnews:novidadesprojects:projetospublications:publicaçõesrepositories:repositóriosteaching:ensinosubmenus:submenusunk:página não encontradaprojects:titles:project1:Projeto 1project2:Projeto 2project3:Projeto 3project4:Projeto 4project5:Projeto 5project6:Projeto 6
Now, all we have to do is open all the files inside _projects/ and change its titles. For the file _projects/1_project.md, for example, will look like this:
title:projects.titles.project1
Finally, we can run our website again and it will work as expected. :tada:.
Now THIS is a translated title.
Don’t forget to do the same for the other pages that use different layouts, like cv. Open the file _layouts/cv.html and modify:
Now, let’s see how to add translated content. For example, let’s say that we want to translate the biography in the about page. The best way to achieve this is to create translated parts of the pages for each language, and then import the correct version. For this, cut the whole biography part of the about page (_pages/about.md), leaving only the header. Then, create a new file _i18n/en/pages/about.md and paste the content there. Do the same for the file _i18n/pt-br/pages/about.md, but adding the translated version. Now, in the file _pages/about.md, add the translate_file function, pointing to the new about files. The final _pages/about.md will look like this:
---
layout: about
title: titles.about
subtitle: <ahref='#'>Affiliations</a>. Address. Contacts. Moto. Etc.
permalink: /
profile:
align: right
image: prof_pic.jpg
image_circular: false # crops the image to make it circular
address: >
<p>555 your office number</p><p>123 your address street</p><p>Your City, State 12345</p>
news: true # includes a list of news items
selected_papers: true # includes a list of papers marked as "selected={true}"
social: true # includes social icons at the bottom of the page
---
{% translate_file pages/about.md %}
Biography in Portuguese.
Fixing page title in the browser
Currently, when you open a section of your site, in the browser tab the section name is not translated. To solve this, open the file _includes/metadata.html and change the following code:
This will trigger another problem, caused by the title of the second news in the about page. We need to add a translated title for this new. Now, do the following changes:
File _news/announcement_2.md:
# title: A long announcement with detailstitle:news.titles.news2
File _i18n/en.yml:
news:titles:news2:A long announcement with details
File _i18n/pt-br.yml:
news:titles:news2:Um anúncio longo com detalhes
Summing up
Adding a translated version of your site is not hard, but it is kind of annoying. Even with the help of this great plugin, there are some little details that, if missed, will take a lot of time and patience to master (at least took me). There are a few things that can be done also, like translating the description of the pages and more of their content, but since these can be done based on the same concepts that I showed here, I’ll leave it as a homework :laughing:. I will detail how to create translated versions of your cv and blog in other posts, since this one is already really long. I hope this post was useful to you. If you have any questions, feel free to ask in the comments.
]]>Running locally your own al-folio website2022-09-27T22:13:16+00:002022-09-27T22:13:16+00:00https://george-gca.github.io/blog/2022/running-local-al-folioThis post is part of a series of posts that explain how to set up your own site based on the al-folio theme and add support for a second language:
I decided to write this post because I had a hard time figuring out how to run my own Jekyll page locally, especially since I don’t regularly use Jekyll. I hope this post will help you to run your own al-folio website locally, or whatever other Jekyll-based theme you have.
Installing Ruby and rbenv
I ran my code in a native Linux environment (Ubuntu 22.04.1 LTS), but I tested it in a WSL environment as well. First things first: you need to install Ruby language support. The recommended way by the al-folio creators is using rbenv. For those familiar with Python, rbenv is similar to pyenv. To install rbenv, run the following commands:
git clone https://github.com/rbenv/rbenv.git ~/.rbenv
cd ~/.rbenv && src/configure && make -C src
This will download the rbenv repository to your home directory, and then compile it. DON’T install it via apt, since it will download an older version of the package, and will not allow you to install the latest Ruby versions. Next, add the following lines to your ~/.bashrc file:
Then, restart your terminal or run . ~/.bashrc to reload your bash settings. This will make the rbenv command available in your terminal. To test if this is working properly, run curl -fsSL https://github.com/rbenv/rbenv-installer/raw/main/bin/rbenv-doctor | bash. It should output something similar to this:
Checking for `rbenv' in PATH: /home/gca/.rbenv/bin/rbenv
Checking for rbenv shims in PATH: Not found
Checking `rbenv install' support: /home/gca/.rbenv/plugins/ruby-build/bin/rbenv-install (ruby-build 20220910.1-10-gecb9d22)
Counting installed Ruby versions: 1 versions
Auditing installed plugins: OK
It will display an error in line Checking for rbenv shims in PATH. Don’t worry, this will be fixed. Next, you need to install ruby-build as a rbenv plugin, so you can easily download and install different versions of Ruby. To do so, run the following commands:
To check which versions of Ruby are available to install, simply run rbenv install --list. You can install any version you want, but I recommend installing the latest stable version. At the time of writing, it is version 3.1.2. To install it, you need first to install the ssl dependency and then the Ruby version.
sudo apt install-y libssl-dev
rbenv install 3.1.2
Installing al-folio dependencies
Now that you have Ruby installed, you can install al-folio dependencies. First, clone the al-folio repository to your local machine. Then, enter the repository directory and create a local Ruby environment with the installed Ruby version. Next, install the bundle package, so it will take care of installing the rest of the dependencies. To do all of this, run the following commands:
git clone ~/git@github.com:alshedivat/al-folio.git
cd ~/al-folio
rbenv local 3.1.2
gem install bundle
bundle install
Since al-folio supports jupyter notebooks, we also need to install it. If you are not planning to use notebooks that much, you can install it via pipx. To install both pipx and jupyter, run the following commands:
# this can also be done via apt with `sudo apt install -y pipx`
python3 -m pip install--user pipx
pipx install jupyter
Running al-folio locally
From now on, you can run the site locally. All you need to do is to open the al-folio directory and call Jekyll:
